[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:18 dist-test-slave-rqvb krb5kdc[2547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:18 dist-test-slave-rqvb krb5kdc[2547](info): set up 2 sockets
Sep 28 02:41:18 dist-test-slave-rqvb krb5kdc[2547](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:41:19.461798 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:19 dist-test-slave-rqvb krb5kdc[2547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:19.585307 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33681
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:19.698261 2560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:19.698546 2560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:19.698586 2560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:19.702695 2560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:19.702770 2560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:19.702790 2560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:19.702806 2560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:19.702821 2560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:19.708371 2560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:19.709481 2560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:19.710526 2560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:19.719437 2565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:19.719488 2568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:19.719563 2566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:19.719676 2560 server_base.cc:1034] running on GCE node
I20260928 02:41:19.720273 2560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:19.721395 2560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:19.722556 2560 hybrid_clock.cc:648] HybridClock initialized: now 1790563279722539 us; error 34 us; skew 500 ppm
Sep 28 02:41:19 dist-test-slave-rqvb krb5kdc[2547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563279, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:19.725445 2560 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:19.726548 2560 webserver.cc:460] Webserver started at http://127.2.123.62:45239/ using document root <none> and password file <none>
I20260928 02:41:19.727128 2560 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:19.727202 2560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:19.727427 2560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:19.729574 2560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "d17e8b3a139844d88f6c528376205fa7"
format_stamp: "Formatted at 2026-09-28 02:41:19 on dist-test-slave-rqvb"
server_key: "8784948122f0c781709e32956fe8e333"
server_key_iv: "bb5d67fa23c3afacd5ca29d40d8a58dc"
server_key_version: "encryptionkey@0"
I20260928 02:41:19.730224 2560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "d17e8b3a139844d88f6c528376205fa7"
format_stamp: "Formatted at 2026-09-28 02:41:19 on dist-test-slave-rqvb"
server_key: "8784948122f0c781709e32956fe8e333"
server_key_iv: "bb5d67fa23c3afacd5ca29d40d8a58dc"
server_key_version: "encryptionkey@0"
I20260928 02:41:19.734170 2560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:41:19.736738 2575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:19.738113 2560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:19.738240 2560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "d17e8b3a139844d88f6c528376205fa7"
format_stamp: "Formatted at 2026-09-28 02:41:19 on dist-test-slave-rqvb"
server_key: "8784948122f0c781709e32956fe8e333"
server_key_iv: "bb5d67fa23c3afacd5ca29d40d8a58dc"
server_key_version: "encryptionkey@0"
I20260928 02:41:19.738348 2560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:19.760908 2560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:19.764503 2560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:19.764724 2560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:19.772032 2560 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:33681
I20260928 02:41:19.772042 2627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:33681 every 8 connection(s)
I20260928 02:41:19.773048 2560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:19.776003 2628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:19.781548 2628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7: Bootstrap starting.
I20260928 02:41:19.782234 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 2560
I20260928 02:41:19.782444 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:19.784096 2540 external_mini_cluster.cc:1428] Setting key adaebeab08daedab5ab418bf45c2c919
I20260928 02:41:19.784730 2628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:19.785672 2628 log.cc:824] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:19.787972 2628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7: No bootstrap required, opened a new log
I20260928 02:41:19.792042 2628 raft_consensus.cc:348] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17e8b3a139844d88f6c528376205fa7" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33681 } }
I20260928 02:41:19.792371 2628 raft_consensus.cc:374] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:19.792438 2628 raft_consensus.cc:729] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17e8b3a139844d88f6c528376205fa7, State: Initialized, Role: FOLLOWER
I20260928 02:41:19.792922 2628 consensus_queue.cc:260] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17e8b3a139844d88f6c528376205fa7" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33681 } }
I20260928 02:41:19.793085 2628 raft_consensus.cc:388] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:19.793166 2628 raft_consensus.cc:482] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:19.793255 2628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:19.794306 2628 raft_consensus.cc:504] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17e8b3a139844d88f6c528376205fa7" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33681 } }
I20260928 02:41:19.794639 2628 leader_election.cc:302] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d17e8b3a139844d88f6c528376205fa7; no voters:
I20260928 02:41:19.794939 2628 leader_election.cc:288] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:19.795025 2633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:19.795192 2633 raft_consensus.cc:686] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [term 1 LEADER]: Becoming Leader. State: Replica: d17e8b3a139844d88f6c528376205fa7, State: Running, Role: LEADER
I20260928 02:41:19.795538 2633 consensus_queue.cc:237] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17e8b3a139844d88f6c528376205fa7" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33681 } }
I20260928 02:41:19.796275 2628 sys_catalog.cc:564] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:19.797205 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d17e8b3a139844d88f6c528376205fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17e8b3a139844d88f6c528376205fa7" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33681 } } }
I20260928 02:41:19.797411 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:19.797689 2635 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d17e8b3a139844d88f6c528376205fa7. Latest consensus state: current_term: 1 leader_uuid: "d17e8b3a139844d88f6c528376205fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17e8b3a139844d88f6c528376205fa7" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33681 } } }
I20260928 02:41:19.797786 2635 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:19.798102 2642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:19.801446 2642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:41:19 dist-test-slave-rqvb krb5kdc[2547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:19.807623 2642 catalog_manager.cc:1347] Generated new cluster ID: 685c127236b049a1ad8d76fce789f9e3
I20260928 02:41:19.807698 2642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:19.811268 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:19.787739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.2.123.62:33681)
0928 02:41:19.788312 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:19.788335 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:19.789486 (+ 1151us) server_negotiation.cc:407] Connection header received
0928 02:41:19.792342 (+ 2856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:19.792359 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:19.792800 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:19.793172 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:19.795914 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.797235 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.798992 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:19.799675 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:19.799840 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:19.804849 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:19.804855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:19.804872 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:19.804906 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:19.807622 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.808287 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.808294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.808300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.808440 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:19.808787 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:19.808791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:19.808793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:19.809198 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:19.809403 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:19.810032 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 02:41:19.810305 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":158,"threads_started":1}
I20260928 02:41:19.837423 2642 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:19.838486 2642 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:19.844853 2642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d17e8b3a139844d88f6c528376205fa7: Generated new TSK 0
I20260928 02:41:19.845603 2642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:19.881605 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33681
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:20.017375 2654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:20.017650 2654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:20.017696 2654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:20.021837 2654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:20.021924 2654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:20.022017 2654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:20.027370 2654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33681
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:20.028484 2654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:20.029615 2654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:20.039108 2660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:20.039113 2659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:20.039113 2662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:20.039399 2654 server_base.cc:1034] running on GCE node
I20260928 02:41:20.039898 2654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:20.040561 2654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:20.041742 2654 hybrid_clock.cc:648] HybridClock initialized: now 1790563280041689 us; error 65 us; skew 500 ppm
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:20.044586 2654 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:20.045476 2654 webserver.cc:460] Webserver started at http://127.2.123.1:43525/ using document root <none> and password file <none>
I20260928 02:41:20.046061 2654 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:20.046134 2654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:20.046304 2654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:20.048364 2654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "86bc2f0e2fe04e7bbd5c2edd4bce5554"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "34aecc57b58fe2b0c7743b0fd26ff2c6"
server_key_iv: "a8a8654b739ba11549b8f79ab9d44390"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.048868 2654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "86bc2f0e2fe04e7bbd5c2edd4bce5554"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "34aecc57b58fe2b0c7743b0fd26ff2c6"
server_key_iv: "a8a8654b739ba11549b8f79ab9d44390"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.052698 2654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:41:20.055182 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.056457 2654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:20.056562 2654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "86bc2f0e2fe04e7bbd5c2edd4bce5554"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "34aecc57b58fe2b0c7743b0fd26ff2c6"
server_key_iv: "a8a8654b739ba11549b8f79ab9d44390"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.056653 2654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:20.078514 2654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:20.081759 2654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:20.081943 2654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:20.082607 2654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:20.083451 2654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:20.083496 2654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.083537 2654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:20.083565 2654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.092128 2654 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:35421
I20260928 02:41:20.092161 2782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:35421 every 8 connection(s)
I20260928 02:41:20.093042 2654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:20.099318 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 2654
I20260928 02:41:20.099424 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:20.099678 2540 external_mini_cluster.cc:1428] Setting key 1e84e67d9fa5c89aed5e1125f845d8ec
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:20.107828 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:20.094860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:54729 (local address 127.2.123.62:33681)
0928 02:41:20.095102 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:20.095106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:20.095977 (+ 871us) server_negotiation.cc:407] Connection header received
0928 02:41:20.097389 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:20.097392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:20.097456 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:20.097529 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:20.098844 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.099445 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.100712 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.101009 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.101101 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:20.103636 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:20.103640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:20.103643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:20.103681 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:20.105528 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.106144 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.106148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.106150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.106248 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.106618 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.106621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.106623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.106859 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:20.106964 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:20.107512 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 02:41:20.107652 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:41:20.108712 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:20.095273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33681 (local address 127.2.123.1:54729)
0928 02:41:20.095741 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:20.095876 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:20.097167 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:20.097705 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:20.097713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:20.098143 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:20.098694 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:20.098710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.099580 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.099583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:20.100606 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:20.100615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.101163 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.101168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:20.101328 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:20.101876 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:20.101892 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:20.103441 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:20.105680 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:20.105686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:20.105700 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:20.106031 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:20.106363 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:20.106365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:20.106367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:20.106519 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:20.106973 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:20.106978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:20.107251 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 02:41:20.107411 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 02:41:20.107701 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260928 02:41:20.109929 2783 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33681
I20260928 02:41:20.110214 2783 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:20.110746 2783 heartbeater.cc:511] Master 127.2.123.62:33681 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:41:20.112577 2592 ts_manager.cc:194] Registered new tserver with Master: 86bc2f0e2fe04e7bbd5c2edd4bce5554 (127.2.123.1:35421)
I20260928 02:41:20.113782 2592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:54729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:20.129359 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33681
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:20.241583 2788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:20.241890 2788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:20.241950 2788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:20.246148 2788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:20.246212 2788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:20.246284 2788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:20.251608 2788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33681
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:20.252665 2788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:20.253711 2788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:20.263377 2793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:20.263336 2794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:20.263412 2788 server_base.cc:1034] running on GCE node
W20260928 02:41:20.263336 2796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:20.263989 2788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:20.264662 2788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:20.265838 2788 hybrid_clock.cc:648] HybridClock initialized: now 1790563280265825 us; error 31 us; skew 500 ppm
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:20.268839 2788 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:20.269742 2788 webserver.cc:460] Webserver started at http://127.2.123.2:40015/ using document root <none> and password file <none>
I20260928 02:41:20.270370 2788 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:20.270439 2788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:20.270663 2788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:20.272801 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "642378a0dfb340f98ead685beccf65a4"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "98d97a101537ab28edfed2c690909a9c"
server_key_iv: "396a272654d53d797136e7566781edba"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.273361 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "642378a0dfb340f98ead685beccf65a4"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "98d97a101537ab28edfed2c690909a9c"
server_key_iv: "396a272654d53d797136e7566781edba"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.277591 2788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 02:41:20.280424 2803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.281857 2788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:20.282008 2788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "642378a0dfb340f98ead685beccf65a4"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "98d97a101537ab28edfed2c690909a9c"
server_key_iv: "396a272654d53d797136e7566781edba"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.282121 2788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:20.300588 2788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:20.303932 2788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:20.304116 2788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:20.304720 2788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:20.305541 2788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:20.305588 2788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.305640 2788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:20.305668 2788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.314514 2788 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:39945
I20260928 02:41:20.314564 2916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:39945 every 8 connection(s)
I20260928 02:41:20.315493 2788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:20.315855 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 2788
I20260928 02:41:20.316005 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:20.316265 2540 external_mini_cluster.cc:1428] Setting key b2f3503a3f1d8102c7d4f8ecbabab0b6
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:41:20.331211 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:20.317408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:60751 (local address 127.2.123.62:33681)
0928 02:41:20.317546 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:20.317550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:20.318752 (+ 1202us) server_negotiation.cc:407] Connection header received
0928 02:41:20.320041 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:20.320044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:20.320105 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:20.320189 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:20.321426 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.322252 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.323500 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.323864 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.323990 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:20.326904 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:20.326910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:20.326913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:20.326945 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:20.328624 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.329267 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.329270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.329271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.329364 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.329818 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.329822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.329858 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.330191 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:20.330301 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:20.330970 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 02:41:20.331083 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:41:20.332250 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:20.317939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33681 (local address 127.2.123.2:60751)
0928 02:41:20.318488 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:20.318543 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:20.319822 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:20.320399 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:20.320406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:20.320780 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:20.321276 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:20.321290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.322378 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.322381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:20.323364 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:20.323374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.324016 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.324019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:20.324165 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:20.324800 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:20.324817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:20.326698 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:20.328746 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:20.328752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:20.328767 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:20.329143 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:20.329458 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:20.329460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:20.329461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:20.329659 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:20.330316 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:20.330324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:20.330679 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 02:41:20.330905 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 02:41:20.331269 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":143,"threads_started":1}
I20260928 02:41:20.333391 2917 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33681
I20260928 02:41:20.333622 2917 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:20.334200 2917 heartbeater.cc:511] Master 127.2.123.62:33681 requested a full tablet report, sending...
I20260928 02:41:20.335285 2592 ts_manager.cc:194] Registered new tserver with Master: 642378a0dfb340f98ead685beccf65a4 (127.2.123.2:39945)
I20260928 02:41:20.335855 2592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:60751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:20.345851 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33681
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:20.457209 2922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:20.457489 2922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:20.457535 2922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:20.461696 2922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:20.461757 2922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:20.461839 2922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:20.467171 2922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33681
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:20.468231 2922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:20.469230 2922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:20.478482 2927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:20.478503 2930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:20.478471 2928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:20.478794 2922 server_base.cc:1034] running on GCE node
I20260928 02:41:20.479245 2922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:20.479867 2922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:20.481047 2922 hybrid_clock.cc:648] HybridClock initialized: now 1790563280481023 us; error 34 us; skew 500 ppm
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:20.484011 2922 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:20.484987 2922 webserver.cc:460] Webserver started at http://127.2.123.3:43177/ using document root <none> and password file <none>
I20260928 02:41:20.485553 2922 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:20.485615 2922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:20.485780 2922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:20.487926 2922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "0df18d88bc464ba289be113b2976a431"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "51b307b916ad98c1bbf70b501b2adc52"
server_key_iv: "6260943daf278be810407a52f0a7b0dd"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.488436 2922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "0df18d88bc464ba289be113b2976a431"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "51b307b916ad98c1bbf70b501b2adc52"
server_key_iv: "6260943daf278be810407a52f0a7b0dd"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.492139 2922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:41:20.494531 2937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.495752 2922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:20.495877 2922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "0df18d88bc464ba289be113b2976a431"
format_stamp: "Formatted at 2026-09-28 02:41:20 on dist-test-slave-rqvb"
server_key: "51b307b916ad98c1bbf70b501b2adc52"
server_key_iv: "6260943daf278be810407a52f0a7b0dd"
server_key_version: "encryptionkey@0"
I20260928 02:41:20.495971 2922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:20.524132 2922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:20.527436 2922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:20.527623 2922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:20.528240 2922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:20.529068 2922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:20.529114 2922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.529155 2922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:20.529182 2922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:20.538597 2922 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:36729
I20260928 02:41:20.538615 3050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:36729 every 8 connection(s)
I20260928 02:41:20.539500 2922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:20.542438 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 2922
I20260928 02:41:20.542572 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:20.542829 2540 external_mini_cluster.cc:1428] Setting key 7b992d933c87b2eb91dd217a3100f678
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:20.552732 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:20.541366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:32921 (local address 127.2.123.62:33681)
0928 02:41:20.541495 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:20.541499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:20.542450 (+ 951us) server_negotiation.cc:407] Connection header received
0928 02:41:20.543977 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:20.543980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:20.544033 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:20.544102 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:20.545255 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.545782 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.546949 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.547170 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.547255 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:20.549606 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:20.549609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:20.549611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:20.549635 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:20.550805 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.551256 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.551260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.551261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.551333 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.551632 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.551635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.551637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.551809 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:20.551965 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:20.552456 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 02:41:20.552572 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:41:20.553519 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:20.541742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33681 (local address 127.2.123.3:32921)
0928 02:41:20.542252 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:20.542296 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:20.543774 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:20.544280 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:20.544291 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:20.544642 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:20.545115 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:20.545129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.545897 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.545901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:20.546829 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:20.546838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.547300 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.547305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:20.547476 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:20.548007 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:20.548022 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:20.549446 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:20.550908 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:20.550913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:20.550929 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:20.551167 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:20.551415 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:20.551418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:20.551420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:20.551547 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:20.551974 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:20.551981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:20.552224 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 02:41:20.552393 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 02:41:20.552712 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":158,"threads_started":1}
I20260928 02:41:20.554709 3051 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33681
I20260928 02:41:20.554955 3051 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:20.555441 3051 heartbeater.cc:511] Master 127.2.123.62:33681 requested a full tablet report, sending...
I20260928 02:41:20.556526 2592 ts_manager.cc:194] Registered new tserver with Master: 0df18d88bc464ba289be113b2976a431 (127.2.123.3:36729)
I20260928 02:41:20.556926 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:20.557130 2592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:32921
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:20.582042 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:20.572903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.2.123.1:35421)
0928 02:41:20.573209 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:20.573222 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:20.573247 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:41:20.573297 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:20.573302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:20.573519 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:20.573639 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:20.574336 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.575117 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.576035 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.576441 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.576528 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:20.578810 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:20.578820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:20.578834 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:20.578860 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:20.580265 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.580683 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.580689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.580691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.580776 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.581030 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.581033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.581035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.581280 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:20.581493 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:20.581673 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:41:20.581825 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":124,"threads_started":1}
W20260928 02:41:20.583138 2697 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40806
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[2547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:20.608183 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:20.600215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.2.123.1:35421)
0928 02:41:20.600359 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:20.600364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:20.600554 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:41:20.600683 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:20.600686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:20.600731 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:20.600812 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:20.601499 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.602090 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.602932 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:20.603149 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:20.603221 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:20.605534 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:20.605537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:20.605539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:20.605564 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:20.606833 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.607195 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.607198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.607199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.607264 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:20.607469 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:20.607471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:20.607472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:20.607619 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:20.607710 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:20.607947 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 02:41:20.608040 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:41:20.609193 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 2654
I20260928 02:41:20.614210 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 2788
I20260928 02:41:20.620164 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 2922
I20260928 02:41:20.624627 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 2560
2026-09-28T02:41:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2144 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[3065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[3065](info): set up 2 sockets
Sep 28 02:41:20 dist-test-slave-rqvb krb5kdc[3065](info): commencing operation
krb5kdc: starting...
W20260928 02:41:21.581346 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:21 dist-test-slave-rqvb krb5kdc[3065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:21.697211 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33883
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:45037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:33883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:21.805155 3078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:21.805465 3078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:21.805526 3078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:21.809664 3078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:21.809734 3078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:21.809767 3078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:21.809790 3078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:21.809819 3078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:21.815197 3078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:45037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:33883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:21.816239 3078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:21.817246 3078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:21.825767 3083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:21.825783 3086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:21.825779 3084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:21.825839 3078 server_base.cc:1034] running on GCE node
I20260928 02:41:21.826666 3078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:21.827769 3078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:21.828985 3078 hybrid_clock.cc:648] HybridClock initialized: now 1790563281828962 us; error 37 us; skew 500 ppm
Sep 28 02:41:21 dist-test-slave-rqvb krb5kdc[3065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:21.831928 3078 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:21.832883 3078 webserver.cc:460] Webserver started at http://127.2.123.62:33233/ using document root <none> and password file <none>
I20260928 02:41:21.833427 3078 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:21.833498 3078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:21.833706 3078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:21.835826 3078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "d11b7504b9f8468b857d6146224fb7b1"
format_stamp: "Formatted at 2026-09-28 02:41:21 on dist-test-slave-rqvb"
server_key: "4ebad080f3bbf69f073f4f8a8240dd2e"
server_key_iv: "fb0245af97b52776d60613e2d5d69e0e"
server_key_version: "encryptionkey@0"
I20260928 02:41:21.836380 3078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "d11b7504b9f8468b857d6146224fb7b1"
format_stamp: "Formatted at 2026-09-28 02:41:21 on dist-test-slave-rqvb"
server_key: "4ebad080f3bbf69f073f4f8a8240dd2e"
server_key_iv: "fb0245af97b52776d60613e2d5d69e0e"
server_key_version: "encryptionkey@0"
I20260928 02:41:21.840169 3078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:41:21.842689 3093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:21.843938 3078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:21.844043 3078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "d11b7504b9f8468b857d6146224fb7b1"
format_stamp: "Formatted at 2026-09-28 02:41:21 on dist-test-slave-rqvb"
server_key: "4ebad080f3bbf69f073f4f8a8240dd2e"
server_key_iv: "fb0245af97b52776d60613e2d5d69e0e"
server_key_version: "encryptionkey@0"
I20260928 02:41:21.844134 3078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:21.877921 3078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:21.881142 3078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:21.881306 3078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:21.888116 3078 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:33883
I20260928 02:41:21.888128 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:33883 every 8 connection(s)
I20260928 02:41:21.889124 3078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:21.892038 3146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:21.893453 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 3078
I20260928 02:41:21.893587 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:21.893860 2540 external_mini_cluster.cc:1428] Setting key 6490faaad991dcb52d1565a0a86af704
I20260928 02:41:21.898473 3146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1: Bootstrap starting.
I20260928 02:41:21.901084 3146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:21.902062 3146 log.cc:824] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:41:21 dist-test-slave-rqvb krb5kdc[3065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:21.904567 3146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1: No bootstrap required, opened a new log
I20260928 02:41:21.908499 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:21.895311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.2.123.62:33883)
0928 02:41:21.895709 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:21.895724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:21.895761 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:41:21.896390 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:21.896407 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:21.896765 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:21.897053 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:21.897884 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:21.898915 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:21.900038 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:21.900684 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:21.900810 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:21.903877 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:21.903883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:21.903896 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:21.903990 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:21.905586 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:21.906122 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:21.906126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:21.906131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:21.906256 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:21.906566 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:21.906568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:21.906569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:21.906961 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:21.907094 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:21.907395 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 02:41:21.907654 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":126,"threads_started":1}
I20260928 02:41:21.908707 3146 raft_consensus.cc:348] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b7504b9f8468b857d6146224fb7b1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33883 } }
I20260928 02:41:21.908965 3146 raft_consensus.cc:374] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:21.909030 3146 raft_consensus.cc:729] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11b7504b9f8468b857d6146224fb7b1, State: Initialized, Role: FOLLOWER
I20260928 02:41:21.909508 3146 consensus_queue.cc:260] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b7504b9f8468b857d6146224fb7b1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33883 } }
I20260928 02:41:21.909672 3146 raft_consensus.cc:388] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:21.909751 3146 raft_consensus.cc:482] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:21.909866 3146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:21.910848 3146 raft_consensus.cc:504] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b7504b9f8468b857d6146224fb7b1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33883 } }
I20260928 02:41:21.911201 3146 leader_election.cc:302] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d11b7504b9f8468b857d6146224fb7b1; no voters:
I20260928 02:41:21.911548 3146 leader_election.cc:288] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:21.911654 3151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:21.911868 3151 raft_consensus.cc:686] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [term 1 LEADER]: Becoming Leader. State: Replica: d11b7504b9f8468b857d6146224fb7b1, State: Running, Role: LEADER
I20260928 02:41:21.912245 3151 consensus_queue.cc:237] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b7504b9f8468b857d6146224fb7b1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33883 } }
I20260928 02:41:21.912753 3146 sys_catalog.cc:564] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:21.913676 3152 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d11b7504b9f8468b857d6146224fb7b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b7504b9f8468b857d6146224fb7b1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33883 } } }
I20260928 02:41:21.913807 3152 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:21.914203 3153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d11b7504b9f8468b857d6146224fb7b1. Latest consensus state: current_term: 1 leader_uuid: "d11b7504b9f8468b857d6146224fb7b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b7504b9f8468b857d6146224fb7b1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33883 } } }
I20260928 02:41:21.914280 3153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:21.914565 3160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:21.917706 3160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:21.923178 3160 catalog_manager.cc:1347] Generated new cluster ID: 726a2d05f36d4300854966308ee97252
I20260928 02:41:21.923257 3160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:21.933789 3160 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:21.934955 3160 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:21.940958 3160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d11b7504b9f8468b857d6146224fb7b1: Generated new TSK 0
I20260928 02:41:21.941747 3160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:21.977661 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33883
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:45037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:22.091923 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:22.092213 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:22.092288 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:22.096297 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:22.096367 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:22.096453 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:22.101737 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:45037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:22.103152 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:22.104214 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:22.114172 3172 server_base.cc:1034] running on GCE node
W20260928 02:41:22.114068 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:22.114068 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:22.114077 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:22.114785 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:22.115509 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:22.116695 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790563282116674 us; error 39 us; skew 500 ppm
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:22.119465 3172 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:22.120342 3172 webserver.cc:460] Webserver started at http://127.2.123.1:40409/ using document root <none> and password file <none>
I20260928 02:41:22.120862 3172 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:22.120920 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:22.121083 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:22.123178 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "95d111e532544194946bcad30ecb41a3"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "c7e194090b06aaeac31a152948da288d"
server_key_iv: "61e4303d5674967447a991ce95685371"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.123718 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "95d111e532544194946bcad30ecb41a3"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "c7e194090b06aaeac31a152948da288d"
server_key_iv: "61e4303d5674967447a991ce95685371"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.127501 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:41:22.130020 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.131320 3172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:22.131431 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "95d111e532544194946bcad30ecb41a3"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "c7e194090b06aaeac31a152948da288d"
server_key_iv: "61e4303d5674967447a991ce95685371"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.131544 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:22.146600 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:22.150097 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:22.150281 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:22.150853 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:22.151686 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:22.151733 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.151773 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:22.151803 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.160063 3172 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:33909
I20260928 02:41:22.160156 3300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:33909 every 8 connection(s)
I20260928 02:41:22.161037 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:22.162828 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 3172
I20260928 02:41:22.162935 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:22.163205 2540 external_mini_cluster.cc:1428] Setting key edcbbe23212c80c0e9303f0362f002a7
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:41:22.176771 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.163000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:45765 (local address 127.2.123.62:33883)
0928 02:41:22.163126 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.163130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.164180 (+ 1050us) server_negotiation.cc:407] Connection header received
0928 02:41:22.165882 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.165885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.165936 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.166056 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:22.167438 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.168227 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.169555 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.169931 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.170072 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.172733 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:22.172736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:22.172738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:22.172765 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:22.174289 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.174852 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.174856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.174859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.174958 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.175422 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.175424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.175426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.175638 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:22.175705 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.176416 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.176596 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:41:22.177541 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.163495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33883 (local address 127.2.123.1:45765)
0928 02:41:22.164009 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:22.164077 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:22.165614 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:22.166246 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:22.166256 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:22.166740 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:22.167268 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.167283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.168378 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.168381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.169425 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.169436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.170148 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.170152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.170306 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.170889 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:22.170908 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:22.172546 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:22.174400 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:22.174406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:22.174421 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:22.174740 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:22.175086 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:22.175089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:22.175091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:22.175327 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:22.175745 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:22.175751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:22.176085 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 02:41:22.176316 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 02:41:22.176659 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
I20260928 02:41:22.178797 3301 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33883
I20260928 02:41:22.179042 3301 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:22.179541 3301 heartbeater.cc:511] Master 127.2.123.62:33883 requested a full tablet report, sending...
I20260928 02:41:22.181094 3110 ts_manager.cc:194] Registered new tserver with Master: 95d111e532544194946bcad30ecb41a3 (127.2.123.1:33909)
I20260928 02:41:22.182300 3110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:45765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:22.190665 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33883
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:45037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:22.300625 3306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:22.300911 3306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:22.300956 3306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:22.305058 3306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:22.305133 3306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:22.305195 3306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:22.310566 3306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:45037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:22.311604 3306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:22.312690 3306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:22.322654 3306 server_base.cc:1034] running on GCE node
W20260928 02:41:22.322606 3311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:22.322603 3314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:22.322603 3312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:22.323348 3306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:22.324020 3306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:22.325201 3306 hybrid_clock.cc:648] HybridClock initialized: now 1790563282325180 us; error 42 us; skew 500 ppm
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:22.328115 3306 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:22.329028 3306 webserver.cc:460] Webserver started at http://127.2.123.2:42637/ using document root <none> and password file <none>
I20260928 02:41:22.329576 3306 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:22.329653 3306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:22.329854 3306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:22.331960 3306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "b86a015cb26446788d085c768996ff43"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "53b597447d66ceb20bb1f7f907ffaa72"
server_key_iv: "8404751f52794d754f990efc556ebc68"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.332520 3306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "b86a015cb26446788d085c768996ff43"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "53b597447d66ceb20bb1f7f907ffaa72"
server_key_iv: "8404751f52794d754f990efc556ebc68"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.336388 3306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:41:22.338897 3321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.340169 3306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:22.340278 3306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "b86a015cb26446788d085c768996ff43"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "53b597447d66ceb20bb1f7f907ffaa72"
server_key_iv: "8404751f52794d754f990efc556ebc68"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.340378 3306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:22.361613 3306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:22.364856 3306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:22.365033 3306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:22.365612 3306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:22.366504 3306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:22.366550 3306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.366638 3306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:22.366664 3306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.375319 3306 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:42105
I20260928 02:41:22.375347 3434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:42105 every 8 connection(s)
I20260928 02:41:22.376221 3306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:22.377190 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 3306
I20260928 02:41:22.377286 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:22.377530 2540 external_mini_cluster.cc:1428] Setting key 799fbd6e574ce498219bddd32dd58058
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:41:22.391539 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.378141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:52995 (local address 127.2.123.62:33883)
0928 02:41:22.378279 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.378283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.379347 (+ 1064us) server_negotiation.cc:407] Connection header received
0928 02:41:22.380725 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.380728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.380795 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.380880 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:22.382462 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.383019 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.384759 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.384995 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.385085 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.387938 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:22.387942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:22.387944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:22.387970 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:22.389355 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.389906 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.389910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.389911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.390030 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.390417 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.390421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.390423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.390616 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:22.390728 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.391240 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.391370 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:41:22.392370 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.378600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33883 (local address 127.2.123.2:52995)
0928 02:41:22.379115 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:22.379206 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:22.380516 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:22.381103 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:22.381112 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:22.381606 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:22.382289 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.382305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.383166 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.383170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.384604 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.384615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.385159 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.385164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.385369 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.386092 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:22.386114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:22.387762 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:22.389494 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:22.389500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:22.389517 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:22.389786 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:22.390117 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:22.390121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:22.390123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:22.390312 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:22.390746 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:22.390752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:22.390996 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 02:41:22.391171 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:41:22.391507 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":146,"threads_started":1}
I20260928 02:41:22.393585 3435 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33883
I20260928 02:41:22.393839 3435 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:22.394388 3435 heartbeater.cc:511] Master 127.2.123.62:33883 requested a full tablet report, sending...
I20260928 02:41:22.395541 3110 ts_manager.cc:194] Registered new tserver with Master: b86a015cb26446788d085c768996ff43 (127.2.123.2:42105)
I20260928 02:41:22.396345 3110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:52995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:22.406299 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33883
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:45037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:22.515439 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:22.515712 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:22.515755 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:22.519704 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:22.519757 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:22.519814 3440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:22.525039 3440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:45037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:22.526057 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:22.527002 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:22.536502 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:22.536521 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:22.536485 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:22.536756 3440 server_base.cc:1034] running on GCE node
I20260928 02:41:22.537132 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:22.537778 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:22.538929 3440 hybrid_clock.cc:648] HybridClock initialized: now 1790563282538903 us; error 39 us; skew 500 ppm
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:22.541514 3440 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:22.542434 3440 webserver.cc:460] Webserver started at http://127.2.123.3:35399/ using document root <none> and password file <none>
I20260928 02:41:22.542954 3440 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:22.543018 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:22.543206 3440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:22.545264 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "eb3971d8812142d09562adf76e944bb7"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "099cc3edadd0c383f05ecc8d0f924a51"
server_key_iv: "79d93a1f45399d14175a4e694a23b903"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.545782 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "eb3971d8812142d09562adf76e944bb7"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "099cc3edadd0c383f05ecc8d0f924a51"
server_key_iv: "79d93a1f45399d14175a4e694a23b903"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.549492 3440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:22.551961 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.553242 3440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:22.553352 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "eb3971d8812142d09562adf76e944bb7"
format_stamp: "Formatted at 2026-09-28 02:41:22 on dist-test-slave-rqvb"
server_key: "099cc3edadd0c383f05ecc8d0f924a51"
server_key_iv: "79d93a1f45399d14175a4e694a23b903"
server_key_version: "encryptionkey@0"
I20260928 02:41:22.553448 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:22.571264 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:22.574685 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:22.574873 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:22.575511 3440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:22.576345 3440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:22.576392 3440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.576445 3440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:22.576473 3440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:22.584759 3440 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:43209
I20260928 02:41:22.584787 3568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:43209 every 8 connection(s)
I20260928 02:41:22.585634 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:22.592458 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 3440
I20260928 02:41:22.592579 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:22.592847 2540 external_mini_cluster.cc:1428] Setting key 23b6e9c787fae9a9da74e6a725b8607b
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:22.599291 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.587454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:42253 (local address 127.2.123.62:33883)
0928 02:41:22.587605 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.587609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.588447 (+ 838us) server_negotiation.cc:407] Connection header received
0928 02:41:22.589805 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.589809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.589868 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.589943 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:22.591189 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.591722 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.593236 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.593511 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.593611 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.596076 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:22.596080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:22.596082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:22.596110 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:22.597260 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.597754 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.597757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.597759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.597838 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.598203 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.598206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.598208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.598398 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:22.598484 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.599004 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.599127 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:41:22.600134 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.587859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33883 (local address 127.2.123.3:42253)
0928 02:41:22.588300 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:22.588342 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:22.589582 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:22.590160 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:22.590167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:22.590542 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:22.591032 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.591046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.591841 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.591844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.593104 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.593116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.593667 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.593671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.593857 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.594484 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:22.594500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:22.595909 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:22.597369 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:22.597375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:22.597394 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:22.597657 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:22.597918 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:22.597921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:22.597924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:22.598117 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:22.598495 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:22.598502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:22.598763 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 02:41:22.598928 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:41:22.599277 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260928 02:41:22.601254 3569 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33883
I20260928 02:41:22.601473 3569 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:22.601946 3569 heartbeater.cc:511] Master 127.2.123.62:33883 requested a full tablet report, sending...
I20260928 02:41:22.602952 3110 ts_manager.cc:194] Registered new tserver with Master: eb3971d8812142d09562adf76e944bb7 (127.2.123.3:43209)
I20260928 02:41:22.603499 3110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:42253
I20260928 02:41:22.606629 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:22.616775 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.609761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.2.123.62:33883)
0928 02:41:22.609908 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.609912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.610043 (+ 131us) server_negotiation.cc:407] Connection header received
0928 02:41:22.610162 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.610165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.610204 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.610268 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:22.610859 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.611346 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.612217 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.612418 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.612490 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.613872 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:22.613875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:22.613877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:22.613898 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:22.615254 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.615637 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.615640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.615641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.615733 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.615916 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.615918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.615919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.616120 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:22.616236 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.616488 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.616583 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:41:22.621482 3110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:22.624586 3110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:22.639397 3580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.633270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:33909 (local address 127.0.0.1:39542)
0928 02:41:22.633616 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:22.633669 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:22.633799 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:22.634596 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:22.634602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:22.634624 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:22.634761 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.634769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.636214 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.636217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.637719 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.637730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.638839 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.638843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.638952 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.638989 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:41:22.639117 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:41:22.639243 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":34,"thread_start_us":79,"threads_started":1}
I20260928 02:41:22.639402 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.633454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.2.123.1:33909)
0928 02:41:22.633754 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.633772 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.633798 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:41:22.634009 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.634015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.634252 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.634404 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:22.634889 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.636034 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.637851 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.638739 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.638830 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.638995 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.639156 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.639249 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
I20260928 02:41:22.639894 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.634394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.2.123.3:43209)
0928 02:41:22.634712 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.634724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.635072 (+ 348us) server_negotiation.cc:407] Connection header received
0928 02:41:22.635151 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.635156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.635353 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.635502 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:22.635795 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.636813 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.638281 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.638932 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.639032 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.639163 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.639629 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.639740 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":76,"threads_started":1}
I20260928 02:41:22.640173 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.633883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:42105 (local address 127.0.0.1:48946)
0928 02:41:22.634851 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:22.634863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:22.634976 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:22.635400 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:22.635403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:22.635419 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:22.635529 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.635535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.637091 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.637094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.638561 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.638572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.639779 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.639783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.639883 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.639905 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:41:22.639970 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:41:22.640047 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":32,"spinlock_wait_cycles":3200,"thread_start_us":112,"threads_started":1}
I20260928 02:41:22.640203 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.634047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:43209 (local address 127.0.0.1:35678)
0928 02:41:22.634864 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:22.634875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:22.634976 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:22.635575 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:22.635577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:22.635593 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:22.635680 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.635685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.636964 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.636967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.638172 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.638181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.639225 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.639228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.639322 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.639345 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:41:22.639411 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:41:22.639480 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1,"spinlock_wait_cycles":6656,"thread_start_us":80,"threads_started":1}
I20260928 02:41:22.641017 3584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.634303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.2.123.2:42105)
0928 02:41:22.634712 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.634724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.634967 (+ 243us) server_negotiation.cc:407] Connection header received
0928 02:41:22.635125 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.635130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.635280 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.635404 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:22.635678 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.636675 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.638691 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.639483 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.639573 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.639719 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.640785 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.640876 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":100,"threads_started":1}
I20260928 02:41:22.642798 3235 tablet_service.cc:1467] Processing CreateTablet for tablet fc8374e2c24e447386aea49eab1cc1c8 (DEFAULT_TABLE table=test-table [id=3cb8cbfa57234353a480283272c4a862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:22.643392 3503 tablet_service.cc:1467] Processing CreateTablet for tablet fc8374e2c24e447386aea49eab1cc1c8 (DEFAULT_TABLE table=test-table [id=3cb8cbfa57234353a480283272c4a862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:22.643826 3235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8374e2c24e447386aea49eab1cc1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:22.644433 3503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8374e2c24e447386aea49eab1cc1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:22.644527 3369 tablet_service.cc:1467] Processing CreateTablet for tablet fc8374e2c24e447386aea49eab1cc1c8 (DEFAULT_TABLE table=test-table [id=3cb8cbfa57234353a480283272c4a862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:22.645507 3369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8374e2c24e447386aea49eab1cc1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:22.650182 3586 tablet_bootstrap.cc:492] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3: Bootstrap starting.
I20260928 02:41:22.652437 3586 tablet_bootstrap.cc:654] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:22.652442 3588 tablet_bootstrap.cc:492] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43: Bootstrap starting.
I20260928 02:41:22.652978 3587 tablet_bootstrap.cc:492] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7: Bootstrap starting.
I20260928 02:41:22.653611 3586 log.cc:824] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:22.654999 3587 tablet_bootstrap.cc:654] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:22.654999 3588 tablet_bootstrap.cc:654] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:22.655725 3586 tablet_bootstrap.cc:492] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3: No bootstrap required, opened a new log
I20260928 02:41:22.655849 3588 log.cc:824] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:22.655900 3586 ts_tablet_manager.cc:1397] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:41:22.656141 3587 log.cc:824] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:22.657629 3588 tablet_bootstrap.cc:492] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43: No bootstrap required, opened a new log
I20260928 02:41:22.657804 3588 ts_tablet_manager.cc:1397] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:41:22.658144 3587 tablet_bootstrap.cc:492] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7: No bootstrap required, opened a new log
I20260928 02:41:22.658325 3587 ts_tablet_manager.cc:1397] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260928 02:41:22.659634 3586 raft_consensus.cc:348] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
I20260928 02:41:22.659919 3586 raft_consensus.cc:374] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:22.660005 3586 raft_consensus.cc:729] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d111e532544194946bcad30ecb41a3, State: Initialized, Role: FOLLOWER
I20260928 02:41:22.660476 3586 consensus_queue.cc:260] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
I20260928 02:41:22.661185 3301 heartbeater.cc:503] Master 127.2.123.62:33883 was elected leader, sending a full tablet report...
I20260928 02:41:22.661187 3588 raft_consensus.cc:348] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
I20260928 02:41:22.661423 3588 raft_consensus.cc:374] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:22.661469 3588 raft_consensus.cc:729] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86a015cb26446788d085c768996ff43, State: Initialized, Role: FOLLOWER
I20260928 02:41:22.661490 3586 ts_tablet_manager.cc:1428] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:41:22.661907 3588 consensus_queue.cc:260] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
W20260928 02:41:22.662292 3302 tablet.cc:2369] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:22.662202 3587 raft_consensus.cc:348] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
I20260928 02:41:22.662449 3587 raft_consensus.cc:374] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:22.662499 3587 raft_consensus.cc:729] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3971d8812142d09562adf76e944bb7, State: Initialized, Role: FOLLOWER
I20260928 02:41:22.662889 3587 consensus_queue.cc:260] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
I20260928 02:41:22.662997 3588 ts_tablet_manager.cc:1428] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260928 02:41:22.663321 3435 heartbeater.cc:503] Master 127.2.123.62:33883 was elected leader, sending a full tablet report...
I20260928 02:41:22.663480 3569 heartbeater.cc:503] Master 127.2.123.62:33883 was elected leader, sending a full tablet report...
I20260928 02:41:22.663793 3587 ts_tablet_manager.cc:1428] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260928 02:41:22.730197 3593 raft_consensus.cc:482] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:22.730495 3593 raft_consensus.cc:504] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
I20260928 02:41:22.731902 3593 leader_election.cc:288] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95d111e532544194946bcad30ecb41a3 (127.2.123.1:33909), eb3971d8812142d09562adf76e944bb7 (127.2.123.3:43209)
I20260928 02:41:22.736253 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.732248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:48403 (local address 127.2.123.1:33909)
0928 02:41:22.732386 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.732391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.732407 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:41:22.732479 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.732482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.732529 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.732595 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:22.732960 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.733632 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.735156 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.735701 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.735804 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.735868 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.736039 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.736116 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:41:22.736294 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.732137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:33909 (local address 127.2.123.2:48403)
0928 02:41:22.732257 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:22.732270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:22.732363 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:22.732714 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:22.732717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:22.732737 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:22.732836 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.732841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.733749 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.733752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.735041 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.735052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.735813 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.735816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.735907 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.735932 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:41:22.736013 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:41:22.736160 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260928 02:41:22.736501 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.732458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:43209 (local address 127.2.123.2:34503)
0928 02:41:22.732881 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:22.732896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:22.733009 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:22.733266 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:22.733268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:22.733283 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:22.733390 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.733395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.734209 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.734213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.735331 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:22.735338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.736123 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.736127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:22.736224 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.736247 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:41:22.736322 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:41:22.736383 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":124,"threads_started":1}
I20260928 02:41:22.736536 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.732563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:34503 (local address 127.2.123.3:43209)
0928 02:41:22.732703 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.732707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.732968 (+ 261us) server_negotiation.cc:407] Connection header received
0928 02:41:22.733116 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.733119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.733166 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.733242 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:22.733515 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.734109 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.735438 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.736000 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.736092 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.736146 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.736347 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.736417 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:41:22.737031 3255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8374e2c24e447386aea49eab1cc1c8" candidate_uuid: "b86a015cb26446788d085c768996ff43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d111e532544194946bcad30ecb41a3" is_pre_election: true
I20260928 02:41:22.737214 3523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8374e2c24e447386aea49eab1cc1c8" candidate_uuid: "b86a015cb26446788d085c768996ff43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3971d8812142d09562adf76e944bb7" is_pre_election: true
I20260928 02:41:22.737345 3255 raft_consensus.cc:2454] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b86a015cb26446788d085c768996ff43 in term 0.
I20260928 02:41:22.737483 3523 raft_consensus.cc:2454] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b86a015cb26446788d085c768996ff43 in term 0.
I20260928 02:41:22.737816 3322 leader_election.cc:302] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d111e532544194946bcad30ecb41a3, b86a015cb26446788d085c768996ff43; no voters:
I20260928 02:41:22.738157 3593 raft_consensus.cc:2781] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:22.738231 3593 raft_consensus.cc:482] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:22.738260 3593 raft_consensus.cc:3037] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:22.739226 3593 raft_consensus.cc:504] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
I20260928 02:41:22.739542 3593 leader_election.cc:288] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [CANDIDATE]: Term 1 election: Requested vote from peers 95d111e532544194946bcad30ecb41a3 (127.2.123.1:33909), eb3971d8812142d09562adf76e944bb7 (127.2.123.3:43209)
I20260928 02:41:22.739923 3255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8374e2c24e447386aea49eab1cc1c8" candidate_uuid: "b86a015cb26446788d085c768996ff43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d111e532544194946bcad30ecb41a3"
I20260928 02:41:22.740010 3523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8374e2c24e447386aea49eab1cc1c8" candidate_uuid: "b86a015cb26446788d085c768996ff43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3971d8812142d09562adf76e944bb7"
I20260928 02:41:22.740059 3255 raft_consensus.cc:3037] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:22.740118 3523 raft_consensus.cc:3037] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:22.741317 3523 raft_consensus.cc:2454] T fc8374e2c24e447386aea49eab1cc1c8 P eb3971d8812142d09562adf76e944bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b86a015cb26446788d085c768996ff43 in term 1.
I20260928 02:41:22.741317 3255 raft_consensus.cc:2454] T fc8374e2c24e447386aea49eab1cc1c8 P 95d111e532544194946bcad30ecb41a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b86a015cb26446788d085c768996ff43 in term 1.
I20260928 02:41:22.741604 3322 leader_election.cc:302] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d111e532544194946bcad30ecb41a3, b86a015cb26446788d085c768996ff43; no voters:
I20260928 02:41:22.741760 3593 raft_consensus.cc:2781] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:22.742010 3593 raft_consensus.cc:686] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [term 1 LEADER]: Becoming Leader. State: Replica: b86a015cb26446788d085c768996ff43, State: Running, Role: LEADER
I20260928 02:41:22.742395 3593 consensus_queue.cc:237] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } }
I20260928 02:41:22.745393 3110 catalog_manager.cc:5495] T fc8374e2c24e447386aea49eab1cc1c8 P b86a015cb26446788d085c768996ff43 reported cstate change: term changed from 0 to 1, leader changed from <none> to b86a015cb26446788d085c768996ff43 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "b86a015cb26446788d085c768996ff43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d111e532544194946bcad30ecb41a3" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb3971d8812142d09562adf76e944bb7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b86a015cb26446788d085c768996ff43" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 42105 } health_report { overall_health: HEALTHY } } }
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:22.764175 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.755481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.2.123.1:33909)
0928 02:41:22.755660 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.755666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.755840 (+ 174us) server_negotiation.cc:407] Connection header received
0928 02:41:22.755969 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.755972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.756055 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.756129 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:22.756852 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.757400 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.758325 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.758540 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.758629 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.760914 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:22.760927 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:22.760942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:22.760966 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:22.762569 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.762958 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.762963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.762964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.763056 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.763297 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.763300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.763302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.763542 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:22.763738 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.763902 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.764011 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:22.789455 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.781794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.2.123.1:33909)
0928 02:41:22.781938 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.781942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.782043 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:41:22.782182 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.782184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.782230 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.782299 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:22.782924 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.783457 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.784289 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.784504 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.784599 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.786797 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:22.786800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:22.786802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:22.786827 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:22.788099 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.788470 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.788473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.788474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.788539 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.788747 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.788749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.788750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.788906 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:22.789002 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.789212 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.789310 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:22.813454 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:22.805656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.2.123.1:33909)
0928 02:41:22.805803 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:22.805808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:22.805900 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:41:22.806051 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:22.806054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:22.806106 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:22.806176 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:22.806819 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.807409 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.808197 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:22.808437 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:22.808524 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:22.810697 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:22.810700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:22.810702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:22.810727 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:22.812032 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.812368 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.812370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.812372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.812448 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:22.812657 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:22.812659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:22.812660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:22.812853 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:22.812951 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:22.813224 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:41:22.813331 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 02:41:22.815097 3215 tablet_service.cc:2908] Rejecting scan request for tablet fc8374e2c24e447386aea49eab1cc1c8: Uninitialized: clean time has not yet been initialized
I20260928 02:41:22.816973 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 3172
I20260928 02:41:22.822280 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 3306
I20260928 02:41:22.826930 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 3440
I20260928 02:41:22.831280 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 3078
2026-09-28T02:41:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2206 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3610](info): set up 2 sockets
Sep 28 02:41:22 dist-test-slave-rqvb krb5kdc[3610](info): commencing operation
krb5kdc: starting...
W20260928 02:41:23.785702 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:23 dist-test-slave-rqvb krb5kdc[3610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:23.911571 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33073
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:33073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:24.027736 3623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.028023 3623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.028064 3623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.032157 3623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:24.032229 3623 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:41:24.032245 3623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.032262 3623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.032277 3623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:24.032296 3623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:24.037760 3623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:33073
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:24.038939 3623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.040019 3623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.048708 3631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.048717 3629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.048728 3628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.049068 3623 server_base.cc:1034] running on GCE node
I20260928 02:41:24.049618 3623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.050791 3623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.051990 3623 hybrid_clock.cc:648] HybridClock initialized: now 1790563284051969 us; error 35 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.054911 3623 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:24.055909 3623 webserver.cc:460] Webserver started at http://127.2.123.62:41813/ using document root <none> and password file <none>
I20260928 02:41:24.056458 3623 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.056524 3623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.056766 3623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:24.058893 3623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "b1d1422b37b647358f932a55a8c92efa"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "77545d5ca3e88d46b9c05fe8fcfe4f52"
server_key_iv: "9357101514df39ab29cbb366f7f5d1d5"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.059444 3623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "b1d1422b37b647358f932a55a8c92efa"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "77545d5ca3e88d46b9c05fe8fcfe4f52"
server_key_iv: "9357101514df39ab29cbb366f7f5d1d5"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.063418 3623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:41:24.066030 3638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.067451 3623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:41:24.067567 3623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "b1d1422b37b647358f932a55a8c92efa"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "77545d5ca3e88d46b9c05fe8fcfe4f52"
server_key_iv: "9357101514df39ab29cbb366f7f5d1d5"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.067669 3623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.087863 3623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.091197 3623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.091396 3623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.099355 3623 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:33073
I20260928 02:41:24.099377 3700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:33073 every 8 connection(s)
I20260928 02:41:24.100363 3623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:24.103231 3701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:24.108399 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 3623
I20260928 02:41:24.108515 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:24.108783 2540 external_mini_cluster.cc:1428] Setting key 5d7e777689c2a76c93ea75c2d6d46578
I20260928 02:41:24.109104 3701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa: Bootstrap starting.
I20260928 02:41:24.111647 3701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:24.112542 3701 log.cc:824] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.114645 3701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa: No bootstrap required, opened a new log
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:24.118391 3701 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d1422b37b647358f932a55a8c92efa" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33073 } }
I20260928 02:41:24.118682 3701 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:24.118744 3701 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1d1422b37b647358f932a55a8c92efa, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.119284 3701 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d1422b37b647358f932a55a8c92efa" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33073 } }
I20260928 02:41:24.119454 3701 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:24.119522 3701 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:24.119625 3701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:24.120548 3701 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d1422b37b647358f932a55a8c92efa" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33073 } }
I20260928 02:41:24.120940 3701 leader_election.cc:302] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1d1422b37b647358f932a55a8c92efa; no voters:
I20260928 02:41:24.121323 3701 leader_election.cc:288] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:24.121447 3706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:24.121747 3706 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [term 1 LEADER]: Becoming Leader. State: Replica: b1d1422b37b647358f932a55a8c92efa, State: Running, Role: LEADER
I20260928 02:41:24.122203 3706 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d1422b37b647358f932a55a8c92efa" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33073 } }
I20260928 02:41:24.122733 3701 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:24.123775 3706 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1d1422b37b647358f932a55a8c92efa. Latest consensus state: current_term: 1 leader_uuid: "b1d1422b37b647358f932a55a8c92efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d1422b37b647358f932a55a8c92efa" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33073 } } }
I20260928 02:41:24.124306 3707 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1d1422b37b647358f932a55a8c92efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d1422b37b647358f932a55a8c92efa" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33073 } } }
I20260928 02:41:24.124387 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.110234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.2.123.62:33073)
0928 02:41:24.110661 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.110680 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.110718 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:41:24.111364 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.111382 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.111730 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.112033 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.112892 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.113920 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.115063 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.115609 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.116097 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.118709 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.118716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.118734 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.118771 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.120649 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.121198 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.121205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.121212 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.121385 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.121770 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.121773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.121775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.122357 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.122547 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.122893 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.123209 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
I20260928 02:41:24.124511 3707 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:24.124410 3706 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:24.125489 3718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:24.129570 3718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:24.135423 3718 catalog_manager.cc:1347] Generated new cluster ID: daad0e5508b64b128d3192b878bea5be
I20260928 02:41:24.135519 3718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:24.175854 3718 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:24.176941 3718 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:24.181836 3718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa: Generated new TSK 0
I20260928 02:41:24.182574 3718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:24.216542 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33073
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:24.328572 3727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.328881 3727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.328943 3727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.333161 3727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.333237 3727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.333295 3727 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:24.333354 3727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:24.338984 3727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:24.340109 3727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.341130 3727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.350941 3735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.351032 3727 server_base.cc:1034] running on GCE node
W20260928 02:41:24.350921 3732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.350921 3733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.351521 3727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.352216 3727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.353430 3727 hybrid_clock.cc:648] HybridClock initialized: now 1790563284353377 us; error 37 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.356367 3727 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:24.357259 3727 webserver.cc:460] Webserver started at http://127.2.123.1:41129/ using document root <none> and password file <none>
I20260928 02:41:24.357784 3727 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.357851 3727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.358072 3727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:24.360167 3727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "e79c423935264e4badb3cb1b55d5f65c"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "3940c3ddba828d86c222d0dacda50765"
server_key_iv: "3b224defd7d4e41e1cf2ef8c4c49332f"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.360844 3727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "e79c423935264e4badb3cb1b55d5f65c"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "3940c3ddba828d86c222d0dacda50765"
server_key_iv: "3b224defd7d4e41e1cf2ef8c4c49332f"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.364832 3727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:41:24.367611 3742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.368968 3727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:41:24.369088 3727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "e79c423935264e4badb3cb1b55d5f65c"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "3940c3ddba828d86c222d0dacda50765"
server_key_iv: "3b224defd7d4e41e1cf2ef8c4c49332f"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.369205 3727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.381238 3727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.384465 3727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.384632 3727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.386082 3727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:24.386137 3727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.386188 3727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:24.386217 3727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:24.396045 3727 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:46661
I20260928 02:41:24.396153 3857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:46661 every 8 connection(s)
I20260928 02:41:24.397058 3727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:24.398684 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.386925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:36361 (local address 127.2.123.62:33073)
0928 02:41:24.387068 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.387073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.387937 (+ 864us) server_negotiation.cc:407] Connection header received
0928 02:41:24.389067 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.389071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.389133 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.389238 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.390440 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.390968 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.392068 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.392356 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.392457 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.394980 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.394984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.394987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.395015 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.396596 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.397068 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.397071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.397074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.397160 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.397473 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.397476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.397478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.397705 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.397811 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.398450 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.398562 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:41:24.399430 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.387332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33073 (local address 127.2.123.1:36361)
0928 02:41:24.387772 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.387820 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.388793 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.389408 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.389417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.389758 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.390295 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.390308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.391085 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.391088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.391952 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.391961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.392517 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.392521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.392697 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.393306 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.393323 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.394818 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.396703 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.396707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.396718 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.396969 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.397248 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.397250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.397251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.397382 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.397887 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.397893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.398181 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.398334 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.398576 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260928 02:41:24.400485 3858 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33073
I20260928 02:41:24.400707 3858 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:24.401166 3858 heartbeater.cc:511] Master 127.2.123.62:33073 requested a full tablet report, sending...
I20260928 02:41:24.402586 3655 ts_manager.cc:194] Registered new tserver with Master: e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661)
I20260928 02:41:24.403573 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 3727
I20260928 02:41:24.403673 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:24.403927 2540 external_mini_cluster.cc:1428] Setting key 136ae9f790a8a7ace808faf0e78f2d4f
I20260928 02:41:24.404049 3655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:36361
I20260928 02:41:24.409802 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.402647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:40167 (local address 127.2.123.62:33073)
0928 02:41:24.402769 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.402773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.402973 (+ 200us) server_negotiation.cc:407] Connection header received
0928 02:41:24.403068 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.403071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.403112 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.403189 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.403688 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.404200 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.405263 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.405560 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.405665 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.406875 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.406879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.406882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.406908 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.408264 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.408732 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.408735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.408737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.408809 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.409062 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.409065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.409067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.409231 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.409316 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.409552 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.409651 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:41:24.409811 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.402578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33073 (local address 127.2.123.1:40167)
0928 02:41:24.402885 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.402899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.402996 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.403209 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.403212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.403429 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.403561 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.403568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.404355 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.404358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.405147 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.405157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.405686 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.405690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.405770 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.406297 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.406311 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.406720 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.408361 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.408364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.408366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.408627 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.408901 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.408904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.408906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.408986 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.409317 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.409320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.409407 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.409489 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.409623 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:24.428884 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33073
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:24.538720 3867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.539003 3867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.539047 3867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.543133 3867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.543203 3867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.543249 3867 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:24.543283 3867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:24.548626 3867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:24.549643 3867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.550668 3867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.560539 3873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.560494 3875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.560631 3867 server_base.cc:1034] running on GCE node
W20260928 02:41:24.560503 3872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.561172 3867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.561821 3867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.563005 3867 hybrid_clock.cc:648] HybridClock initialized: now 1790563284562977 us; error 38 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.565863 3867 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:24.566804 3867 webserver.cc:460] Webserver started at http://127.2.123.2:35549/ using document root <none> and password file <none>
I20260928 02:41:24.567335 3867 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.567402 3867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.567618 3867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:24.569782 3867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "f376bc1b6b16436e9a9f467f544cd6b3"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "68121cc9cde6992ff5188326dfc99fa3"
server_key_iv: "5d88597833b033905d7c70b54e839d6f"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.570354 3867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "f376bc1b6b16436e9a9f467f544cd6b3"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "68121cc9cde6992ff5188326dfc99fa3"
server_key_iv: "5d88597833b033905d7c70b54e839d6f"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.574206 3867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:41:24.576565 3882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.577831 3867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:24.577939 3867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "f376bc1b6b16436e9a9f467f544cd6b3"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "68121cc9cde6992ff5188326dfc99fa3"
server_key_iv: "5d88597833b033905d7c70b54e839d6f"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.578064 3867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.599498 3867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.602773 3867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.602955 3867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.604404 3867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:24.604454 3867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.604506 3867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:24.604533 3867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:24.614652 3867 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:38399
I20260928 02:41:24.614720 3997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:38399 every 8 connection(s)
I20260928 02:41:24.615721 3867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:24.618347 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.605255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:44031 (local address 127.2.123.62:33073)
0928 02:41:24.605406 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.605410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.606492 (+ 1082us) server_negotiation.cc:407] Connection header received
0928 02:41:24.607523 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.607526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.607584 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.607666 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.609255 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.609873 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.611567 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.611874 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.611962 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.614828 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.614832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.614835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.614865 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.616417 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.616887 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.616890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.616892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.616971 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.617286 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.617289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.617291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.617475 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.617558 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.618066 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.618185 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:41:24.619089 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.605817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33073 (local address 127.2.123.2:44031)
0928 02:41:24.606320 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.606363 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.607284 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.607885 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.607896 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.608356 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.609066 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.609087 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.610003 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.610007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.611428 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.611439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.612020 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.612024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.612226 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.612870 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.612891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.614611 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.616527 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.616532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.616546 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.616784 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.617052 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.617055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.617057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.617187 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.617568 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.617575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.617821 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.617994 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.618311 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":169,"threads_started":1}
I20260928 02:41:24.620213 3998 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33073
I20260928 02:41:24.620348 3998 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:24.620898 3998 heartbeater.cc:511] Master 127.2.123.62:33073 requested a full tablet report, sending...
I20260928 02:41:24.622030 3655 ts_manager.cc:194] Registered new tserver with Master: f376bc1b6b16436e9a9f467f544cd6b3 (127.2.123.2:38399)
I20260928 02:41:24.622771 3655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:44031
I20260928 02:41:24.625223 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 3867
I20260928 02:41:24.625313 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:24.625535 2540 external_mini_cluster.cc:1428] Setting key 423836e3e7ccb305df32a90cf5e3b589
I20260928 02:41:24.629952 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.622518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:49681 (local address 127.2.123.62:33073)
0928 02:41:24.622639 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.622643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.622870 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:41:24.622975 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.622978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.623024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.623095 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.623559 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.624078 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.625029 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.625251 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.625333 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.626439 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.626442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.626444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.626464 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.628015 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.628542 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.628546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.628548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.628635 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.628941 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.628944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.628945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.629214 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.629290 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.629654 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.629794 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:41:24.630440 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.622446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33073 (local address 127.2.123.2:49681)
0928 02:41:24.622790 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.622802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.622891 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.623128 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.623131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.623347 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.623453 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.623458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.624198 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.624202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.624927 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.624936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.625343 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.625346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.625431 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.625885 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.625899 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.626301 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.628141 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.628144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.628146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.628425 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.628741 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.628743 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.628745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.628841 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.629364 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.629366 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.629462 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.630112 (+ 650us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.630271 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:24.653103 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33073
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:24.762712 4007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:24.762988 4007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:24.763033 4007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:24.767019 4007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:24.767086 4007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:24.767124 4007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:24.767158 4007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:24.772547 4007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:33073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:24.773589 4007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:24.774654 4007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:24.784363 4013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:24.784373 4012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.784516 4007 server_base.cc:1034] running on GCE node
W20260928 02:41:24.784683 4015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:24.785081 4007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:24.785737 4007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:24.786922 4007 hybrid_clock.cc:648] HybridClock initialized: now 1790563284786888 us; error 47 us; skew 500 ppm
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:24.789594 4007 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:24.790529 4007 webserver.cc:460] Webserver started at http://127.2.123.3:41755/ using document root <none> and password file <none>
I20260928 02:41:24.791070 4007 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:24.791136 4007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:24.791334 4007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:24.793396 4007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "054edb5b2841471e94845ccc9021368f"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "c3e1e5257f7570f94e0ac7d63f384835"
server_key_iv: "3b986771a37b20ed04f6de658a28ed37"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.793977 4007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "054edb5b2841471e94845ccc9021368f"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "c3e1e5257f7570f94e0ac7d63f384835"
server_key_iv: "3b986771a37b20ed04f6de658a28ed37"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.797735 4007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:41:24.800312 4022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.801640 4007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:24.801753 4007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "054edb5b2841471e94845ccc9021368f"
format_stamp: "Formatted at 2026-09-28 02:41:24 on dist-test-slave-rqvb"
server_key: "c3e1e5257f7570f94e0ac7d63f384835"
server_key_iv: "3b986771a37b20ed04f6de658a28ed37"
server_key_version: "encryptionkey@0"
I20260928 02:41:24.801853 4007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:24.843931 4007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:24.847344 4007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:24.847523 4007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:24.848964 4007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:24.849014 4007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:24.849066 4007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:24.849094 4007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:24.859643 4007 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:33171
I20260928 02:41:24.859696 4137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:33171 every 8 connection(s)
I20260928 02:41:24.860827 4007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:24.863121 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.849846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:34389 (local address 127.2.123.62:33073)
0928 02:41:24.849991 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.849996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.851125 (+ 1129us) server_negotiation.cc:407] Connection header received
0928 02:41:24.852322 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.852327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.852390 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.852550 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.854044 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.854874 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.856087 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.856409 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.856541 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.859173 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.859178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.859180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.859214 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.860862 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.861488 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.861491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.861493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.861592 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.861929 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.861932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.861933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.862179 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.862262 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.862885 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.862990 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:41:24.863883 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.850465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33073 (local address 127.2.123.3:34389)
0928 02:41:24.850983 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.851027 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.852049 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.852751 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.852760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.853187 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.853860 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.853879 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.855015 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.855018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.855960 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.855969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.856580 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.856583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.856739 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.857388 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.857408 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.858991 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.860991 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.860997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.861013 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.861360 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.861670 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.861673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.861674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.861801 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.862263 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.862267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.862578 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.862776 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.863067 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
I20260928 02:41:24.865085 4138 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33073
I20260928 02:41:24.865226 4138 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:24.865700 4138 heartbeater.cc:511] Master 127.2.123.62:33073 requested a full tablet report, sending...
I20260928 02:41:24.866721 3654 ts_manager.cc:194] Registered new tserver with Master: 054edb5b2841471e94845ccc9021368f (127.2.123.3:33171)
I20260928 02:41:24.867427 3654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:34389
I20260928 02:41:24.869743 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 4007
I20260928 02:41:24.869840 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:24.870152 2540 external_mini_cluster.cc:1428] Setting key e9cbcf0f555f5ad36420edfc1512621f
I20260928 02:41:24.872314 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:24.874981 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.867208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:47931 (local address 127.2.123.62:33073)
0928 02:41:24.867314 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.867318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.867469 (+ 151us) server_negotiation.cc:407] Connection header received
0928 02:41:24.867583 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.867586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.867630 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.867705 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.868325 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.869081 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.870014 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.870303 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.870406 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.871712 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.871716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.871718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.871743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.873230 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.873690 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.873694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.873695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.873782 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.874053 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.874056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.874058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.874280 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.874385 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.874755 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.874865 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:41:24.875027 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.867141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33073 (local address 127.2.123.3:47931)
0928 02:41:24.867398 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.867409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.867499 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.867939 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.867942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.868102 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:24.868214 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.868219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.869218 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.869221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.869853 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.869864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.870454 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.870467 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.870558 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.871112 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:24.871128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:24.871589 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:24.873403 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.873406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.873407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.873602 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.873858 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:24.873860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:24.873861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:24.873929 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:24.874419 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:24.874423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:24.874543 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.874664 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.874832 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:41:24 dist-test-slave-rqvb krb5kdc[3610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:24.898249 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.890162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.2.123.62:33073)
0928 02:41:24.890353 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.890356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.890370 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:41:24.890411 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.890413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.890453 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.890527 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:24.891263 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.891785 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.892595 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.892827 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.892926 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.895316 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:24.895319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:24.895322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:24.895349 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:24.896692 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.897099 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.897102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.897104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.897180 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:24.897417 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:24.897420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:24.897422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:24.897602 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:24.897714 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.897983 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.898094 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:41:24.901268 3654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:24.904366 3654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:24.917999 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.912274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:33171 (local address 127.0.0.1:54506)
0928 02:41:24.912598 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.912654 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.912783 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.913629 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.913634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.913653 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:24.913816 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.913825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.915392 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.915395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.916464 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.916473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.917479 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.917482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.917566 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.917597 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.917698 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.917815 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260928 02:41:24.918042 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.912285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46661 (local address 127.0.0.1:45588)
0928 02:41:24.913115 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.913128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.913213 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.913890 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.913893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.913910 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:24.914029 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.914036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.915438 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.915441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.916729 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.916740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.917712 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.917715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.917792 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.917811 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.917857 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.917910 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":608,"thread_start_us":103,"threads_started":1}
I20260928 02:41:24.918035 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.912348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.2.123.3:33171)
0928 02:41:24.913117 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.913127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.913150 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:41:24.913227 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.913232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.913465 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.913983 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:24.914113 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.915278 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.916586 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.917341 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.917446 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.917604 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.917743 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.917839 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":119,"threads_started":1}
I20260928 02:41:24.918664 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.912724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.2.123.1:46661)
0928 02:41:24.913058 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.913064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.913310 (+ 246us) server_negotiation.cc:407] Connection header received
0928 02:41:24.913387 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.913392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.913579 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.913748 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:24.914146 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.915278 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.916852 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.917594 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.917700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.917830 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.917896 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.918567 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":93,"threads_started":1}
I20260928 02:41:24.919456 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.913232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:38399 (local address 127.0.0.1:42070)
0928 02:41:24.913430 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:24.913443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:24.913523 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:24.914396 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:24.914399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:24.914414 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:24.914491 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.914497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.915430 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.915433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.916729 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:24.916740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.919047 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.919051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:24.919153 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.919176 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:41:24.919253 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:41:24.919325 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":106,"thread_start_us":83,"threads_started":1}
I20260928 02:41:24.919886 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:24.913621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.2.123.2:38399)
0928 02:41:24.914031 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:24.914046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:24.914066 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:41:24.914133 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:24.914138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:24.914284 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:24.914398 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:24.914571 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.915303 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.918029 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:24.918750 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:24.918845 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:24.918985 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:24.919609 (+ 624us) server_negotiation.cc:299] Negotiation successful
0928 02:41:24.919736 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":99,"threads_started":1}
I20260928 02:41:24.921301 3792 tablet_service.cc:1467] Processing CreateTablet for tablet 2b86f171b510426a9ed628d36a87f11d (DEFAULT_TABLE table=test-table [id=2458ce177a384776af8ca5b55e597321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:24.921308 4072 tablet_service.cc:1467] Processing CreateTablet for tablet 2b86f171b510426a9ed628d36a87f11d (DEFAULT_TABLE table=test-table [id=2458ce177a384776af8ca5b55e597321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:24.922449 3792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b86f171b510426a9ed628d36a87f11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:24.922461 4072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b86f171b510426a9ed628d36a87f11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:24.928534 3932 tablet_service.cc:1467] Processing CreateTablet for tablet 2b86f171b510426a9ed628d36a87f11d (DEFAULT_TABLE table=test-table [id=2458ce177a384776af8ca5b55e597321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:24.929440 3932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b86f171b510426a9ed628d36a87f11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:24.929910 4161 tablet_bootstrap.cc:492] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: Bootstrap starting.
I20260928 02:41:24.931242 4162 tablet_bootstrap.cc:492] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Bootstrap starting.
I20260928 02:41:24.932212 4161 tablet_bootstrap.cc:654] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:24.932888 4162 tablet_bootstrap.cc:654] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:24.933140 4161 log.cc:824] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.933624 4162 log.cc:824] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.935231 4162 tablet_bootstrap.cc:492] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: No bootstrap required, opened a new log
I20260928 02:41:24.935232 4161 tablet_bootstrap.cc:492] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: No bootstrap required, opened a new log
I20260928 02:41:24.935410 4162 ts_tablet_manager.cc:1397] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:41:24.935410 4161 ts_tablet_manager.cc:1397] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:41:24.937093 4165 tablet_bootstrap.cc:492] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Bootstrap starting.
I20260928 02:41:24.938808 4162 raft_consensus.cc:348] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:24.939088 4162 raft_consensus.cc:374] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:24.939172 4162 raft_consensus.cc:729] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054edb5b2841471e94845ccc9021368f, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.939406 4165 tablet_bootstrap.cc:654] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:24.939364 4161 raft_consensus.cc:348] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:24.939610 4161 raft_consensus.cc:374] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:24.939668 4161 raft_consensus.cc:729] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79c423935264e4badb3cb1b55d5f65c, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.939638 4162 consensus_queue.cc:260] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:24.940311 4138 heartbeater.cc:503] Master 127.2.123.62:33073 was elected leader, sending a full tablet report...
I20260928 02:41:24.940678 4162 ts_tablet_manager.cc:1428] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260928 02:41:24.940953 4165 log.cc:824] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:24.940132 4161 consensus_queue.cc:260] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:24.942126 3858 heartbeater.cc:503] Master 127.2.123.62:33073 was elected leader, sending a full tablet report...
I20260928 02:41:24.942385 4161 ts_tablet_manager.cc:1428] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260928 02:41:24.944813 4165 tablet_bootstrap.cc:492] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: No bootstrap required, opened a new log
I20260928 02:41:24.945036 4165 ts_tablet_manager.cc:1397] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 02:41:24.948107 4165 raft_consensus.cc:348] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:24.948338 4165 raft_consensus.cc:374] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:24.948370 4165 raft_consensus.cc:729] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f376bc1b6b16436e9a9f467f544cd6b3, State: Initialized, Role: FOLLOWER
I20260928 02:41:24.948769 4165 consensus_queue.cc:260] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:24.949535 3998 heartbeater.cc:503] Master 127.2.123.62:33073 was elected leader, sending a full tablet report...
I20260928 02:41:24.949684 4165 ts_tablet_manager.cc:1428] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:41:25.048208 4169 raft_consensus.cc:482] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:25.048408 4169 raft_consensus.cc:504] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.049495 4169 leader_election.cc:288] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 054edb5b2841471e94845ccc9021368f (127.2.123.3:33171), e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661)
I20260928 02:41:25.053309 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.049899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:57793 (local address 127.2.123.3:33171)
0928 02:41:25.050044 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.050049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.050389 (+ 340us) server_negotiation.cc:407] Connection header received
0928 02:41:25.050467 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.050470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.050525 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.050596 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:25.050839 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.051349 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.052404 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.052812 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.052907 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.052971 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.053134 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.053201 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:41:25.053332 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.049791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:33171 (local address 127.2.123.2:57793)
0928 02:41:25.050262 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.050276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:25.050386 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:25.050617 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:25.050620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:25.050633 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:25.050730 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.050734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.051449 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.051452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.052289 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.052298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.052921 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.052925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.053027 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.053046 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:41:25.053109 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:41:25.053176 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":37,"thread_start_us":156,"threads_started":1}
I20260928 02:41:25.053871 4092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b86f171b510426a9ed628d36a87f11d" candidate_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "054edb5b2841471e94845ccc9021368f" is_pre_election: true
I20260928 02:41:25.053984 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.049791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46661 (local address 127.2.123.2:37753)
0928 02:41:25.050033 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.050059 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:25.050185 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:25.050694 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:25.050696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:25.050711 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:25.050792 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.050796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.051569 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.051572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.052804 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.052814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.053552 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.053555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.053656 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.053676 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:41:25.053745 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:41:25.053828 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":23}
I20260928 02:41:25.053990 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.049900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:37753 (local address 127.2.123.1:46661)
0928 02:41:25.050044 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.050049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.050140 (+ 91us) server_negotiation.cc:407] Connection header received
0928 02:41:25.050312 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.050315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.050381 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.050449 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:25.050928 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.051456 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.052924 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.053341 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.053432 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.053497 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.053779 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.053856 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:41:25.054206 4092 raft_consensus.cc:2454] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f376bc1b6b16436e9a9f467f544cd6b3 in term 0.
I20260928 02:41:25.054601 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b86f171b510426a9ed628d36a87f11d" candidate_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79c423935264e4badb3cb1b55d5f65c" is_pre_election: true
I20260928 02:41:25.054719 3884 leader_election.cc:302] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054edb5b2841471e94845ccc9021368f, f376bc1b6b16436e9a9f467f544cd6b3; no voters:
I20260928 02:41:25.054903 3812 raft_consensus.cc:2454] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f376bc1b6b16436e9a9f467f544cd6b3 in term 0.
I20260928 02:41:25.055017 4169 raft_consensus.cc:2781] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:25.055115 4169 raft_consensus.cc:482] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:25.055171 4169 raft_consensus.cc:3037] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.056239 4169 raft_consensus.cc:504] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.056535 4169 leader_election.cc:288] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [CANDIDATE]: Term 1 election: Requested vote from peers 054edb5b2841471e94845ccc9021368f (127.2.123.3:33171), e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661)
I20260928 02:41:25.056826 4092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b86f171b510426a9ed628d36a87f11d" candidate_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "054edb5b2841471e94845ccc9021368f"
I20260928 02:41:25.056914 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b86f171b510426a9ed628d36a87f11d" candidate_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79c423935264e4badb3cb1b55d5f65c"
I20260928 02:41:25.056947 4092 raft_consensus.cc:3037] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.057029 3812 raft_consensus.cc:3037] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.058025 3812 raft_consensus.cc:2454] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f376bc1b6b16436e9a9f467f544cd6b3 in term 1.
I20260928 02:41:25.058250 4092 raft_consensus.cc:2454] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f376bc1b6b16436e9a9f467f544cd6b3 in term 1.
I20260928 02:41:25.058312 3885 leader_election.cc:302] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e79c423935264e4badb3cb1b55d5f65c, f376bc1b6b16436e9a9f467f544cd6b3; no voters:
I20260928 02:41:25.058459 4169 raft_consensus.cc:2781] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:25.058676 4169 raft_consensus.cc:686] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 LEADER]: Becoming Leader. State: Replica: f376bc1b6b16436e9a9f467f544cd6b3, State: Running, Role: LEADER
I20260928 02:41:25.059010 4169 consensus_queue.cc:237] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.062115 3655 catalog_manager.cc:5495] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f376bc1b6b16436e9a9f467f544cd6b3 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } health_report { overall_health: HEALTHY } } }
W20260928 02:41:25.111850 4139 tablet.cc:2369] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:25.113086 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.108847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.2.123.2:38399)
0928 02:41:25.108979 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.108983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.108999 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:41:25.109191 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.109194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.109259 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.109362 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:25.109598 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.110397 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.111317 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.111680 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.111772 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.112183 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:25.112683 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:25.112796 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.112897 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.112955 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:41:25.116878 3999 tablet.cc:2369] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:25.119081 3812 raft_consensus.cc:1261] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Refusing update from remote peer f376bc1b6b16436e9a9f467f544cd6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:25.119081 4092 raft_consensus.cc:1261] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Refusing update from remote peer f376bc1b6b16436e9a9f467f544cd6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:25.119724 4169 consensus_queue.cc:1035] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:25.119951 4171 consensus_queue.cc:1035] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:25.127460 4176 mvcc.cc:205] Tried to move back new op lower bound from 7334147215839764480 to 7334147215604015104. Current Snapshot: MvccSnapshot[applied={T|T < 7334147215839764480}]
I20260928 02:41:25.142256 3654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:25.142596 3654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:25.145815 3654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa: Sending DeleteTablet for 3 replicas of tablet 2b86f171b510426a9ed628d36a87f11d
I20260928 02:41:25.146721 4072 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b86f171b510426a9ed628d36a87f11d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:25 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:54506
I20260928 02:41:25.146938 3932 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b86f171b510426a9ed628d36a87f11d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:25 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:42070
I20260928 02:41:25.146938 3792 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b86f171b510426a9ed628d36a87f11d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:25 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:45588
I20260928 02:41:25.147969 4184 tablet_replica.cc:330] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: stopping tablet replica
I20260928 02:41:25.148356 4184 raft_consensus.cc:2229] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:25.148766 4184 raft_consensus.cc:2258] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:25.149085 3654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:25.149580 3654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:25.150679 4184 ts_tablet_manager.cc:1905] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:25.153389 4184 ts_tablet_manager.cc:1918] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:25.153566 4184 log.cc:1196] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/2b86f171b510426a9ed628d36a87f11d
I20260928 02:41:25.153913 4184 ts_tablet_manager.cc:1939] T 2b86f171b510426a9ed628d36a87f11d P 054edb5b2841471e94845ccc9021368f: Deleting consensus metadata
I20260928 02:41:25.155185 4072 tablet_service.cc:1467] Processing CreateTablet for tablet 159636e2d32344fa8df1a5da1e0d6586 (DEFAULT_TABLE table=test-table [id=b22cbf8dbe244a358650964abfc63397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:25.155222 3640 catalog_manager.cc:4841] TS 054edb5b2841471e94845ccc9021368f (127.2.123.3:33171): tablet 2b86f171b510426a9ed628d36a87f11d (table test-table [id=2458ce177a384776af8ca5b55e597321]) successfully deleted
I20260928 02:41:25.155303 3792 tablet_service.cc:1467] Processing CreateTablet for tablet 159636e2d32344fa8df1a5da1e0d6586 (DEFAULT_TABLE table=test-table [id=b22cbf8dbe244a358650964abfc63397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:25.155445 4072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159636e2d32344fa8df1a5da1e0d6586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:25.155536 3792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159636e2d32344fa8df1a5da1e0d6586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:25.157598 4162 tablet_bootstrap.cc:492] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: Bootstrap starting.
I20260928 02:41:25.157850 4161 tablet_bootstrap.cc:492] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c: Bootstrap starting.
I20260928 02:41:25.158352 3932 tablet_service.cc:1467] Processing CreateTablet for tablet 159636e2d32344fa8df1a5da1e0d6586 (DEFAULT_TABLE table=test-table [id=b22cbf8dbe244a358650964abfc63397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:25.158533 4162 tablet_bootstrap.cc:654] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:25.158615 3932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159636e2d32344fa8df1a5da1e0d6586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:25.158802 4161 tablet_bootstrap.cc:654] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:25.160322 4161 tablet_bootstrap.cc:492] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c: No bootstrap required, opened a new log
I20260928 02:41:25.160413 4161 ts_tablet_manager.cc:1397] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:25.160643 4186 tablet_replica.cc:330] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: stopping tablet replica
I20260928 02:41:25.160851 4165 tablet_bootstrap.cc:492] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: Bootstrap starting.
I20260928 02:41:25.160974 4161 raft_consensus.cc:348] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.161090 4161 raft_consensus.cc:374] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:25.161123 4161 raft_consensus.cc:729] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79c423935264e4badb3cb1b55d5f65c, State: Initialized, Role: FOLLOWER
I20260928 02:41:25.161247 4161 consensus_queue.cc:260] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.161737 4165 tablet_bootstrap.cc:654] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:25.162068 4161 ts_tablet_manager.cc:1428] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:25.160982 4186 raft_consensus.cc:2229] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:25.162487 4186 raft_consensus.cc:2258] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:25.163295 4165 tablet_bootstrap.cc:492] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: No bootstrap required, opened a new log
I20260928 02:41:25.163375 4165 ts_tablet_manager.cc:1397] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:41:25.163810 4165 raft_consensus.cc:348] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.163938 4165 raft_consensus.cc:374] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:25.163973 4165 raft_consensus.cc:729] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f376bc1b6b16436e9a9f467f544cd6b3, State: Initialized, Role: FOLLOWER
I20260928 02:41:25.164095 4165 consensus_queue.cc:260] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.164350 4165 ts_tablet_manager.cc:1428] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:25.164433 4186 ts_tablet_manager.cc:1905] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:25.168563 4186 ts_tablet_manager.cc:1918] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:25.168668 4186 log.cc:1196] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/2b86f171b510426a9ed628d36a87f11d
I20260928 02:41:25.168972 4186 ts_tablet_manager.cc:1939] T 2b86f171b510426a9ed628d36a87f11d P e79c423935264e4badb3cb1b55d5f65c: Deleting consensus metadata
I20260928 02:41:25.170195 4185 tablet_replica.cc:330] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: stopping tablet replica
I20260928 02:41:25.170252 3641 catalog_manager.cc:4841] TS e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661): tablet 2b86f171b510426a9ed628d36a87f11d (table test-table [id=2458ce177a384776af8ca5b55e597321]) successfully deleted
I20260928 02:41:25.170740 4185 raft_consensus.cc:2229] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:25.171272 4162 tablet_bootstrap.cc:492] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: No bootstrap required, opened a new log
I20260928 02:41:25.171375 4162 ts_tablet_manager.cc:1397] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20260928 02:41:25.171849 4162 raft_consensus.cc:348] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.171981 4162 raft_consensus.cc:374] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:25.172021 4162 raft_consensus.cc:729] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054edb5b2841471e94845ccc9021368f, State: Initialized, Role: FOLLOWER
I20260928 02:41:25.172139 4162 consensus_queue.cc:260] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.172331 4185 raft_consensus.cc:2258] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:25.172413 4162 ts_tablet_manager.cc:1428] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:25.174219 4185 ts_tablet_manager.cc:1905] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:25.177472 4185 ts_tablet_manager.cc:1918] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:25.177639 4185 log.cc:1196] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/2b86f171b510426a9ed628d36a87f11d
I20260928 02:41:25.178012 4185 ts_tablet_manager.cc:1939] T 2b86f171b510426a9ed628d36a87f11d P f376bc1b6b16436e9a9f467f544cd6b3: Deleting consensus metadata
I20260928 02:41:25.179291 3641 catalog_manager.cc:4841] TS f376bc1b6b16436e9a9f467f544cd6b3 (127.2.123.2:38399): tablet 2b86f171b510426a9ed628d36a87f11d (table test-table [id=2458ce177a384776af8ca5b55e597321]) successfully deleted
I20260928 02:41:25.239341 4166 raft_consensus.cc:482] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:25.239507 4166 raft_consensus.cc:504] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.240818 4166 leader_election.cc:288] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661), f376bc1b6b16436e9a9f467f544cd6b3 (127.2.123.2:38399)
I20260928 02:41:25.247584 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.241214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46661 (local address 127.2.123.3:37299)
0928 02:41:25.241889 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.241905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:25.242137 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:25.243739 (+ 1602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:25.243742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:25.243761 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:25.243923 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.243932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.244925 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.244929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.246143 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.246153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.247144 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.247148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.247244 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.247266 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:41:25.247333 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:41:25.247407 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":7}
I20260928 02:41:25.248317 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.243198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:58149 (local address 127.2.123.2:38399)
0928 02:41:25.243345 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.243350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.243368 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:41:25.243436 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.243439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.243491 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.243592 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:25.244560 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.245346 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.246911 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.247536 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.247611 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.247672 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.248105 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.248195 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:41:25.248473 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.243025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:37299 (local address 127.2.123.1:46661)
0928 02:41:25.243328 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.243333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.243348 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:41:25.243396 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.243399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.243457 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.243529 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:25.244060 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.244793 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.246294 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.246911 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.247015 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.247085 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.248256 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.248342 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260928 02:41:25.248842 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.241488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:38399 (local address 127.2.123.3:58149)
0928 02:41:25.242596 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.242610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:25.242701 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:25.243726 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:25.243730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:25.243749 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:25.244396 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.244405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.245483 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.245486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.246752 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.246763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.247854 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.247858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.247973 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.247995 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:41:25.248609 (+ 614us) client_negotiation.cc:240] Negotiation successful
0928 02:41:25.248686 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":488,"thread_start_us":82,"threads_started":1}
I20260928 02:41:25.248863 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159636e2d32344fa8df1a5da1e0d6586" candidate_uuid: "054edb5b2841471e94845ccc9021368f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79c423935264e4badb3cb1b55d5f65c" is_pre_election: true
I20260928 02:41:25.249027 3812 raft_consensus.cc:2454] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 054edb5b2841471e94845ccc9021368f in term 0.
I20260928 02:41:25.249428 4025 leader_election.cc:302] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054edb5b2841471e94845ccc9021368f, e79c423935264e4badb3cb1b55d5f65c; no voters:
I20260928 02:41:25.249500 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159636e2d32344fa8df1a5da1e0d6586" candidate_uuid: "054edb5b2841471e94845ccc9021368f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" is_pre_election: true
I20260928 02:41:25.249704 4166 raft_consensus.cc:2781] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:25.249758 4166 raft_consensus.cc:482] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:25.249733 3952 raft_consensus.cc:2454] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 054edb5b2841471e94845ccc9021368f in term 0.
I20260928 02:41:25.249794 4166 raft_consensus.cc:3037] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.283011 4166 raft_consensus.cc:504] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.283561 4166 leader_election.cc:288] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [CANDIDATE]: Term 1 election: Requested vote from peers e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661), f376bc1b6b16436e9a9f467f544cd6b3 (127.2.123.2:38399)
I20260928 02:41:25.284475 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159636e2d32344fa8df1a5da1e0d6586" candidate_uuid: "054edb5b2841471e94845ccc9021368f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79c423935264e4badb3cb1b55d5f65c"
I20260928 02:41:25.284617 3812 raft_consensus.cc:3037] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.285832 3812 raft_consensus.cc:2454] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 054edb5b2841471e94845ccc9021368f in term 1.
I20260928 02:41:25.286273 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159636e2d32344fa8df1a5da1e0d6586" candidate_uuid: "054edb5b2841471e94845ccc9021368f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f376bc1b6b16436e9a9f467f544cd6b3"
I20260928 02:41:25.286434 3952 raft_consensus.cc:3037] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.287557 3952 raft_consensus.cc:2454] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 054edb5b2841471e94845ccc9021368f in term 1.
I20260928 02:41:25.288070 4025 leader_election.cc:302] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054edb5b2841471e94845ccc9021368f, f376bc1b6b16436e9a9f467f544cd6b3; no voters:
I20260928 02:41:25.288498 4166 raft_consensus.cc:2781] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:25.288722 4166 raft_consensus.cc:686] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 1 LEADER]: Becoming Leader. State: Replica: 054edb5b2841471e94845ccc9021368f, State: Running, Role: LEADER
I20260928 02:41:25.289074 4166 consensus_queue.cc:237] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } }
I20260928 02:41:25.291769 3654 catalog_manager.cc:5495] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f reported cstate change: term changed from 0 to 1, leader changed from <none> to 054edb5b2841471e94845ccc9021368f (127.2.123.3). New cstate: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } health_report { overall_health: HEALTHY } } }
Sep 28 02:41:25 dist-test-slave-rqvb krb5kdc[3610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:25.364843 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.355808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33073 (local address 127.0.0.1:46790)
0928 02:41:25.356159 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.356171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:25.356288 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:25.356555 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:25.356558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:25.357177 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:25.357284 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.357291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.358075 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.358078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.358663 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.358672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.359106 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.359110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.359189 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.359714 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:25.359726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:25.361182 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:25.363062 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:25.363071 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:25.363076 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:25.363399 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:25.363701 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:25.363704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:25.363705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:25.363807 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:25.364235 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:25.364240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:25.364430 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 02:41:25.364538 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:41:25.364689 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20260928 02:41:25.364951 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.355873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.2.123.62:33073)
0928 02:41:25.356077 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.356083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.356245 (+ 162us) server_negotiation.cc:407] Connection header received
0928 02:41:25.356383 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.356386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.356428 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.356495 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:25.357399 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.357945 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.358767 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.358993 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.359082 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.361313 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:25.361317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:25.361319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:25.361343 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:25.362911 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:25.363495 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:25.363498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:25.363500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:25.363586 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:25.363887 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:25.363889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:25.363892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:25.364113 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:25.364203 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.364607 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.364765 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":73}
I20260928 02:41:25.369081 3654 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:46790:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:41:25.369839 3654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:25.374783 3792 tablet_service.cc:1467] Processing CreateTablet for tablet a0e234b428394bb8967426a0959ae4c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da04f5bf49b34d5c84bff1c9106bad79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:25.374895 3932 tablet_service.cc:1467] Processing CreateTablet for tablet a0e234b428394bb8967426a0959ae4c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da04f5bf49b34d5c84bff1c9106bad79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:25.374845 4072 tablet_service.cc:1467] Processing CreateTablet for tablet a0e234b428394bb8967426a0959ae4c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da04f5bf49b34d5c84bff1c9106bad79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:25.375238 3792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e234b428394bb8967426a0959ae4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:25.375244 3932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e234b428394bb8967426a0959ae4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:25.375244 4072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e234b428394bb8967426a0959ae4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:25.377816 4161 tablet_bootstrap.cc:492] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: Bootstrap starting.
I20260928 02:41:25.378033 4162 tablet_bootstrap.cc:492] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: Bootstrap starting.
I20260928 02:41:25.378216 4165 tablet_bootstrap.cc:492] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: Bootstrap starting.
I20260928 02:41:25.378772 4161 tablet_bootstrap.cc:654] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:25.378909 4162 tablet_bootstrap.cc:654] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:25.378917 4165 tablet_bootstrap.cc:654] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:25.379861 4161 tablet_bootstrap.cc:492] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: No bootstrap required, opened a new log
I20260928 02:41:25.379940 4161 ts_tablet_manager.cc:1397] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:25.380388 4165 tablet_bootstrap.cc:492] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: No bootstrap required, opened a new log
I20260928 02:41:25.380471 4165 ts_tablet_manager.cc:1397] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:25.380563 4161 raft_consensus.cc:348] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.380683 4161 raft_consensus.cc:374] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:25.380709 4161 raft_consensus.cc:729] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79c423935264e4badb3cb1b55d5f65c, State: Initialized, Role: FOLLOWER
I20260928 02:41:25.380816 4161 consensus_queue.cc:260] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.380829 4165 raft_consensus.cc:348] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.380939 4165 raft_consensus.cc:374] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:25.380970 4165 raft_consensus.cc:729] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f376bc1b6b16436e9a9f467f544cd6b3, State: Initialized, Role: FOLLOWER
I20260928 02:41:25.381104 4161 ts_tablet_manager.cc:1428] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:25.381074 4165 consensus_queue.cc:260] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.381326 4165 ts_tablet_manager.cc:1428] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:25.381292 4169 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.381410 4169 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:25.381908 4168 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.382076 4168 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:25.382483 4162 tablet_bootstrap.cc:492] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: No bootstrap required, opened a new log
I20260928 02:41:25.382560 4162 ts_tablet_manager.cc:1397] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 02:41:25.382995 4162 raft_consensus.cc:348] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.383100 4162 raft_consensus.cc:374] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:25.383133 4162 raft_consensus.cc:729] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054edb5b2841471e94845ccc9021368f, State: Initialized, Role: FOLLOWER
I20260928 02:41:25.383253 4162 consensus_queue.cc:260] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.383525 4162 ts_tablet_manager.cc:1428] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:25.383657 4166 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.383793 4166 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:41:25.399235 3859 tablet.cc:2369] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:25.691423 4166 raft_consensus.cc:482] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:25.691583 4166 raft_consensus.cc:504] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.692003 4166 leader_election.cc:288] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661), f376bc1b6b16436e9a9f467f544cd6b3 (127.2.123.2:38399)
I20260928 02:41:25.692585 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e234b428394bb8967426a0959ae4c6" candidate_uuid: "054edb5b2841471e94845ccc9021368f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79c423935264e4badb3cb1b55d5f65c" is_pre_election: true
I20260928 02:41:25.692586 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e234b428394bb8967426a0959ae4c6" candidate_uuid: "054edb5b2841471e94845ccc9021368f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" is_pre_election: true
I20260928 02:41:25.692749 3812 raft_consensus.cc:2454] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 054edb5b2841471e94845ccc9021368f in term 0.
I20260928 02:41:25.692749 3952 raft_consensus.cc:2454] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 054edb5b2841471e94845ccc9021368f in term 0.
I20260928 02:41:25.693076 4025 leader_election.cc:302] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054edb5b2841471e94845ccc9021368f, f376bc1b6b16436e9a9f467f544cd6b3; no voters:
I20260928 02:41:25.693284 4166 raft_consensus.cc:2781] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:25.693349 4166 raft_consensus.cc:482] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:25.693393 4166 raft_consensus.cc:3037] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.742980 4191 consensus_queue.cc:1035] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:25.747061 4203 consensus_queue.cc:1035] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:25.770780 4166 raft_consensus.cc:504] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.771309 4166 leader_election.cc:288] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [CANDIDATE]: Term 1 election: Requested vote from peers e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661), f376bc1b6b16436e9a9f467f544cd6b3 (127.2.123.2:38399)
I20260928 02:41:25.771852 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e234b428394bb8967426a0959ae4c6" candidate_uuid: "054edb5b2841471e94845ccc9021368f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79c423935264e4badb3cb1b55d5f65c"
I20260928 02:41:25.771852 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e234b428394bb8967426a0959ae4c6" candidate_uuid: "054edb5b2841471e94845ccc9021368f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f376bc1b6b16436e9a9f467f544cd6b3"
I20260928 02:41:25.772007 3812 raft_consensus.cc:3037] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.772007 3952 raft_consensus.cc:3037] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:25.773236 3812 raft_consensus.cc:2454] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 054edb5b2841471e94845ccc9021368f in term 1.
I20260928 02:41:25.773236 3952 raft_consensus.cc:2454] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 054edb5b2841471e94845ccc9021368f in term 1.
I20260928 02:41:25.773592 4025 leader_election.cc:302] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054edb5b2841471e94845ccc9021368f, f376bc1b6b16436e9a9f467f544cd6b3; no voters:
I20260928 02:41:25.773815 4166 raft_consensus.cc:2781] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:25.773921 4166 raft_consensus.cc:686] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [term 1 LEADER]: Becoming Leader. State: Replica: 054edb5b2841471e94845ccc9021368f, State: Running, Role: LEADER
I20260928 02:41:25.774096 4166 consensus_queue.cc:237] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } }
I20260928 02:41:25.774765 4203 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: TxnStatusTablet state changed. Reason: New leader 054edb5b2841471e94845ccc9021368f. Latest consensus state: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.774885 4203 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:25.775123 4210 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:41:25.775316 4210 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:41:25.775831 3651 catalog_manager.cc:5495] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f reported cstate change: term changed from 0 to 1, leader changed from <none> to 054edb5b2841471e94845ccc9021368f (127.2.123.3). New cstate: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:25.834826 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.830786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.2.123.3:33171)
0928 02:41:25.831115 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.831122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.831140 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:41:25.831203 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.831205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.831255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.831449 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:25.831696 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.832251 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.833121 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.833356 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.833444 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.833742 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:25.834322 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:25.834446 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.834574 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.834652 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260928 02:41:25.834810 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.830665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:33171 (local address 127.0.0.1:54516)
0928 02:41:25.830875 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.830888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:25.830984 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:25.831472 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:25.831475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:25.831484 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:25.831587 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.831593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.832369 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.832373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.832999 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.833014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.833460 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.833464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.833545 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.833590 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:25.834432 (+ 842us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:25.834449 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:41:25.834532 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:41:25.834640 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 02:41:25.837865 3812 raft_consensus.cc:1261] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Refusing update from remote peer 054edb5b2841471e94845ccc9021368f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:25.837909 3952 raft_consensus.cc:1261] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 FOLLOWER]: Refusing update from remote peer 054edb5b2841471e94845ccc9021368f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:25.838380 4166 consensus_queue.cc:1035] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:25.838544 4203 consensus_queue.cc:1035] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:25.840061 4169 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: TxnStatusTablet state changed. Reason: New leader 054edb5b2841471e94845ccc9021368f. Latest consensus state: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.840196 4169 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:25.840148 4168 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: TxnStatusTablet state changed. Reason: New leader 054edb5b2841471e94845ccc9021368f. Latest consensus state: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.840260 4168 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:25.842448 4205 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.842589 4205 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:25.842710 4205 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.842778 4205 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P 054edb5b2841471e94845ccc9021368f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:25.842865 4204 mvcc.cc:205] Tried to move back new op lower bound from 7334147218787631104 to 7334147218531790848. Current Snapshot: MvccSnapshot[applied={T|T < 7334147218787631104}]
I20260928 02:41:25.842965 4168 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.843077 4168 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P e79c423935264e4badb3cb1b55d5f65c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:25.844411 4169 tablet_replica.cc:439] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "054edb5b2841471e94845ccc9021368f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79c423935264e4badb3cb1b55d5f65c" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46661 } } peers { permanent_uuid: "054edb5b2841471e94845ccc9021368f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33171 } } peers { permanent_uuid: "f376bc1b6b16436e9a9f467f544cd6b3" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38399 } } }
I20260928 02:41:25.844527 4169 tablet_replica.cc:442] T a0e234b428394bb8967426a0959ae4c6 P f376bc1b6b16436e9a9f467f544cd6b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:25.848251 4206 mvcc.cc:205] Tried to move back new op lower bound from 7334147218787631104 to 7334147218531790848. Current Snapshot: MvccSnapshot[applied={T|T < 7334147218787631104}]
I20260928 02:41:25.852448 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.848838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.2.123.3:33171)
0928 02:41:25.848972 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.848976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.849145 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:41:25.849263 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.849266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.849317 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.849396 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:25.849585 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.850418 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.851250 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.851588 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.851684 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.851920 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:25.852026 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:25.852131 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.852253 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.852307 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:41:25.862874 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.860041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:40213 (local address 127.2.123.3:33171)
0928 02:41:25.860163 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.860167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.860179 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:25.860285 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.860288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.860335 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.860390 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:25.860602 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.861085 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.861870 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.862148 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.862225 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.862484 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:25.862554 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:25.862619 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.862687 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.862730 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":53}
I20260928 02:41:25.862888 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.859787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:33171 (local address 127.2.123.3:40213)
0928 02:41:25.860088 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.860103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:25.860200 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:25.860403 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:25.860406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:25.860412 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:25.860521 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.860526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.861175 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.861178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.861773 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:25.861780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.862261 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.862264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:25.862338 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.862373 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:25.862618 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:25.862623 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:41:25.862681 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:41:25.862749 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":17,"thread_start_us":93,"threads_started":1}
I20260928 02:41:25.916303 3655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:25.916501 3655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:25.918920 3655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1d1422b37b647358f932a55a8c92efa: Sending DeleteTablet for 3 replicas of tablet 159636e2d32344fa8df1a5da1e0d6586
I20260928 02:41:25.919555 3792 tablet_service.cc:1514] Processing DeleteTablet for tablet 159636e2d32344fa8df1a5da1e0d6586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:25 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:45588
I20260928 02:41:25.919555 3932 tablet_service.cc:1514] Processing DeleteTablet for tablet 159636e2d32344fa8df1a5da1e0d6586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:25 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:42070
I20260928 02:41:25.919595 4071 tablet_service.cc:1514] Processing DeleteTablet for tablet 159636e2d32344fa8df1a5da1e0d6586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:25 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:54506
I20260928 02:41:25.919998 4225 tablet_replica.cc:330] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: stopping tablet replica
I20260928 02:41:25.920012 4227 tablet_replica.cc:330] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: stopping tablet replica
I20260928 02:41:25.920147 4225 raft_consensus.cc:2229] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:25.920183 4227 raft_consensus.cc:2229] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:25.920348 4226 tablet_replica.cc:330] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c: stopping tablet replica
I20260928 02:41:25.920459 4226 raft_consensus.cc:2229] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:25.920574 4226 raft_consensus.cc:2258] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:25.920754 4227 raft_consensus.cc:2258] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:25.920761 4225 raft_consensus.cc:2258] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:25.921502 4227 ts_tablet_manager.cc:1905] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:25.921854 4225 ts_tablet_manager.cc:1905] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:25.923588 4226 ts_tablet_manager.cc:1905] T 159636e2d32344fa8df1a5da1e0d6586 P e79c423935264e4badb3cb1b55d5f65c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:41:25 dist-test-slave-rqvb krb5kdc[3610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563284, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:25.934672 4229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.921248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.2.123.1:46661)
0928 02:41:25.923379 (+ 2131us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.923383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.924209 (+ 826us) server_negotiation.cc:407] Connection header received
0928 02:41:25.924351 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.924354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.924409 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.924548 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:25.925405 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.926256 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.927093 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.927434 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.927534 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.930802 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:25.930816 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:25.930832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:25.930862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:25.932691 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:25.933106 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:25.933114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:25.933116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:25.933234 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:25.933546 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:25.933549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:25.933551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:25.933862 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:25.934114 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.934300 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.934481 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2037,"thread_start_us":74,"threads_started":1}
W20260928 02:41:25.935173 3852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45592
I20260928 02:41:25.944774 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.937067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.2.123.62:33073)
0928 02:41:25.937317 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:25.937322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:25.937413 (+ 91us) server_negotiation.cc:407] Connection header received
0928 02:41:25.937547 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:25.937551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:25.937604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:25.937686 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:25.938448 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.939018 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.939831 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:25.940078 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:25.940178 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:25.941560 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:25.941565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:25.941568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:25.941598 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:25.943060 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:25.943521 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:25.943525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:25.943527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:25.943616 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:25.943880 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:25.943884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:25.943886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:25.944087 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:25.944221 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:25.944456 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 02:41:25.944584 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
W20260928 02:41:25.945298 3655 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46802
I20260928 02:41:25.946940 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 3727
I20260928 02:41:25.977766 4227 ts_tablet_manager.cc:1918] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:25.977766 4225 ts_tablet_manager.cc:1918] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:25.977880 4227 log.cc:1196] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/159636e2d32344fa8df1a5da1e0d6586
I20260928 02:41:25.977881 4225 log.cc:1196] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/159636e2d32344fa8df1a5da1e0d6586
I20260928 02:41:25.978155 4227 ts_tablet_manager.cc:1939] T 159636e2d32344fa8df1a5da1e0d6586 P 054edb5b2841471e94845ccc9021368f: Deleting consensus metadata
I20260928 02:41:25.978159 4225 ts_tablet_manager.cc:1939] T 159636e2d32344fa8df1a5da1e0d6586 P f376bc1b6b16436e9a9f467f544cd6b3: Deleting consensus metadata
W20260928 02:41:25.983213 3641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:46661 (error 108)
I20260928 02:41:25.984413 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 3867
I20260928 02:41:25.984460 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:25.984075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46661 (local address 127.0.0.1:45596)
0928 02:41:25.984282 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:25.984359 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":55,"threads_started":1}
W20260928 02:41:25.985284 3641 catalog_manager.cc:4584] TS e79c423935264e4badb3cb1b55d5f65c (127.2.123.1:46661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 159636e2d32344fa8df1a5da1e0d6586: Network error: Client connection negotiation failed: client connection to 127.2.123.1:46661: connect: Connection refused (error 111)
I20260928 02:41:26.035215 3640 catalog_manager.cc:4841] TS 054edb5b2841471e94845ccc9021368f (127.2.123.3:33171): tablet 159636e2d32344fa8df1a5da1e0d6586 (table test-table [id=b22cbf8dbe244a358650964abfc63397]) successfully deleted
I20260928 02:41:26.038450 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.038158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46661 (local address 127.0.0.1:45612)
0928 02:41:26.038297 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:26.038357 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:41:26.039279 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.039061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46661 (local address 127.0.0.1:45624)
0928 02:41:26.039164 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:26.039213 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:41:26.040518 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 4007
I20260928 02:41:26.041369 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:26.040702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:38399 (local address 127.0.0.1:42090)
0928 02:41:26.041245 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:26.041296 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.2:38399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
I20260928 02:41:26.046187 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 3623
2026-09-28T02:41:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3227 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:28 dist-test-slave-rqvb krb5kdc[4238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:28 dist-test-slave-rqvb krb5kdc[4238](info): set up 2 sockets
Sep 28 02:41:28 dist-test-slave-rqvb krb5kdc[4238](info): commencing operation
krb5kdc: starting...
W20260928 02:41:29.033666 2540 mini_kdc.cc:121] Time spent starting KDC: real 2.954s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:29 dist-test-slave-rqvb krb5kdc[4238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:29.157094 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:39873
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:29.266256 4251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:29.266562 4251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:29.266604 4251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:29.270613 4251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:29.270673 4251 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:41:29.270689 4251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:29.270704 4251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:29.270718 4251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:29.270732 4251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:29.276006 4251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:39873
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:29.277031 4251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:29.278020 4251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:29.286320 4257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:29.286346 4256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:29.286320 4259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:29.286419 4251 server_base.cc:1034] running on GCE node
I20260928 02:41:29.287107 4251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:29.288173 4251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:29.289323 4251 hybrid_clock.cc:648] HybridClock initialized: now 1790563289289300 us; error 40 us; skew 500 ppm
Sep 28 02:41:29 dist-test-slave-rqvb krb5kdc[4238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:29.292012 4251 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:29.292905 4251 webserver.cc:460] Webserver started at http://127.2.123.62:34169/ using document root <none> and password file <none>
I20260928 02:41:29.293452 4251 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:29.293520 4251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:29.293727 4251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:29.295817 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "e93de74285734d19a771ca3065a6fc7f"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "a34677a9e7e2ee20814b93b8ecbbf160"
server_key_iv: "761a21ac9c1cd929dd86e937494f1136"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.296382 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "e93de74285734d19a771ca3065a6fc7f"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "a34677a9e7e2ee20814b93b8ecbbf160"
server_key_iv: "761a21ac9c1cd929dd86e937494f1136"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.300236 4251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:41:29.302772 4266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:29.304105 4251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:29.304224 4251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "e93de74285734d19a771ca3065a6fc7f"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "a34677a9e7e2ee20814b93b8ecbbf160"
server_key_iv: "761a21ac9c1cd929dd86e937494f1136"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.304325 4251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:29.320560 4251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:29.323745 4251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:29.323925 4251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:29.331924 4251 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:39873
I20260928 02:41:29.331943 4328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:39873 every 8 connection(s)
I20260928 02:41:29.332890 4251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:29.333439 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 4251
I20260928 02:41:29.333552 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:29.333807 2540 external_mini_cluster.cc:1428] Setting key 896c5d83cdc8c40aab61b992c691db4a
I20260928 02:41:29.336035 4329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:29.341588 4329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: Bootstrap starting.
Sep 28 02:41:29 dist-test-slave-rqvb krb5kdc[4238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:29.344317 4329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:29.345638 4329 log.cc:824] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:29.347923 4329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: No bootstrap required, opened a new log
I20260928 02:41:29.348135 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.335228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.2.123.62:39873)
0928 02:41:29.335669 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:29.335687 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:29.335724 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:41:29.336461 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:29.336480 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:29.336816 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:29.337147 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:29.337939 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.338977 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.339803 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.340275 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.340420 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.343016 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:29.343023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:29.343038 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:29.343073 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:29.345059 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.345516 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.345522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.345529 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.345672 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.345955 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.345980 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.345982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.346432 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:29.346631 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:29.346864 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 02:41:29.347174 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":124,"threads_started":1}
I20260928 02:41:29.351058 4329 raft_consensus.cc:348] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93de74285734d19a771ca3065a6fc7f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39873 } }
I20260928 02:41:29.351285 4329 raft_consensus.cc:374] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:29.351320 4329 raft_consensus.cc:729] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93de74285734d19a771ca3065a6fc7f, State: Initialized, Role: FOLLOWER
I20260928 02:41:29.351688 4329 consensus_queue.cc:260] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93de74285734d19a771ca3065a6fc7f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39873 } }
I20260928 02:41:29.351796 4329 raft_consensus.cc:388] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:29.351840 4329 raft_consensus.cc:482] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:29.351908 4329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:29.352789 4329 raft_consensus.cc:504] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93de74285734d19a771ca3065a6fc7f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39873 } }
I20260928 02:41:29.353084 4329 leader_election.cc:302] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e93de74285734d19a771ca3065a6fc7f; no voters:
I20260928 02:41:29.353353 4329 leader_election.cc:288] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:29.353466 4334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:29.353675 4334 raft_consensus.cc:686] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [term 1 LEADER]: Becoming Leader. State: Replica: e93de74285734d19a771ca3065a6fc7f, State: Running, Role: LEADER
I20260928 02:41:29.354080 4334 consensus_queue.cc:237] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93de74285734d19a771ca3065a6fc7f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39873 } }
I20260928 02:41:29.354615 4329 sys_catalog.cc:564] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:29.355757 4335 sys_catalog.cc:455] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e93de74285734d19a771ca3065a6fc7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93de74285734d19a771ca3065a6fc7f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39873 } } }
I20260928 02:41:29.355795 4336 sys_catalog.cc:455] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e93de74285734d19a771ca3065a6fc7f. Latest consensus state: current_term: 1 leader_uuid: "e93de74285734d19a771ca3065a6fc7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93de74285734d19a771ca3065a6fc7f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39873 } } }
I20260928 02:41:29.355873 4335 sys_catalog.cc:458] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:29.355891 4336 sys_catalog.cc:458] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:29.356215 4342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:29.359031 4342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:29.364279 4342 catalog_manager.cc:1347] Generated new cluster ID: ab351b41d68e44318e03baf4111f80c1
I20260928 02:41:29.364362 4342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:29.376736 4342 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:29.377704 4342 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:29.387555 4342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: Generated new TSK 0
I20260928 02:41:29.388284 4342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:29.416458 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39873
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:29.528224 4355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:29.528487 4355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:29.528537 4355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:29.532632 4355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:29.532689 4355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:29.532728 4355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:41:29.532742 4355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:29.532794 4355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:29.538094 4355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:29.539254 4355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:29.540304 4355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:29.550217 4360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:29.550206 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:29.550206 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:29.550585 4355 server_base.cc:1034] running on GCE node
I20260928 02:41:29.551082 4355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:29.551764 4355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:29.553012 4355 hybrid_clock.cc:648] HybridClock initialized: now 1790563289553007 us; error 49 us; skew 500 ppm
Sep 28 02:41:29 dist-test-slave-rqvb krb5kdc[4238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:29.556084 4355 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:29.557044 4355 webserver.cc:460] Webserver started at http://127.2.123.1:35637/ using document root <none> and password file <none>
I20260928 02:41:29.557603 4355 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:29.557669 4355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:29.557857 4355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:29.560000 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "f9454dc25771d50c053abbb42a681c39"
server_key_iv: "234e409e686f9c50df426e7777aa0cfb"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.560578 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "f9454dc25771d50c053abbb42a681c39"
server_key_iv: "234e409e686f9c50df426e7777aa0cfb"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.564440 4355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:41:29.566888 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:29.568138 4355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 02:41:29.568253 4355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "f9454dc25771d50c053abbb42a681c39"
server_key_iv: "234e409e686f9c50df426e7777aa0cfb"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.568352 4355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:29.586110 4355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:29.589187 4355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:29.589371 4355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:29.590857 4355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:29.590906 4355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:29.590965 4355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:29.590993 4355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:29 dist-test-slave-rqvb krb5kdc[4238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:29.601286 4355 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:46055
I20260928 02:41:29.601320 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:46055 every 8 connection(s)
I20260928 02:41:29.602381 4355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:29.603061 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 4355
I20260928 02:41:29.603186 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:29.603489 2540 external_mini_cluster.cc:1428] Setting key d36f67e87d5bff262f10919e00423613
I20260928 02:41:29.605124 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.591689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:48735 (local address 127.2.123.62:39873)
0928 02:41:29.591823 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:29.591828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:29.592896 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 02:41:29.593980 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:29.593985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:29.594051 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:29.594108 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:29.595617 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.596201 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.597908 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.598297 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.598429 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.601061 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:29.601066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:29.601069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:29.601101 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:29.602852 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.603431 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.603435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.603437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.603524 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.603857 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.603860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.603861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.604083 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:29.604175 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:29.604789 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 02:41:29.604972 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:41:29.605921 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.592235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39873 (local address 127.2.123.1:48735)
0928 02:41:29.592755 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:29.592799 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:29.593696 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:29.594540 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:29.594550 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:29.594888 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:29.595474 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:29.595489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.596333 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.596337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:29.597778 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:29.597790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.598499 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.598503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:29.598689 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.599312 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:29.599334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:29.600866 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:29.602979 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:29.602985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:29.603001 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:29.603318 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:29.603623 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:29.603626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:29.603628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:29.603763 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:29.604198 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:29.604204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:29.604532 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 02:41:29.604731 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 02:41:29.605057 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":139,"threads_started":1}
I20260928 02:41:29.607184 4486 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39873
I20260928 02:41:29.607321 4486 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:29.607805 4486 heartbeater.cc:511] Master 127.2.123.62:39873 requested a full tablet report, sending...
I20260928 02:41:29.609360 4283 ts_manager.cc:194] Registered new tserver with Master: 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055)
I20260928 02:41:29.610827 4283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:48735
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:41:29.616510 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.609309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39873 (local address 127.2.123.1:48723)
0928 02:41:29.609565 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:29.609578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:29.609693 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:29.609900 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:29.609903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:29.610136 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:29.610231 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:29.610237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.611031 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.611034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:29.611664 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:29.611674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.612317 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.612320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:29.612407 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.612925 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:29.612938 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:29.613335 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:29.615144 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:29.615147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:29.615149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:29.615333 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:29.615593 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:29.615595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:29.615596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:29.615667 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:29.616051 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:29.616053 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:29.616143 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 02:41:29.616234 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:41:29.616347 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260928 02:41:29.616518 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.609243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:48723 (local address 127.2.123.62:39873)
0928 02:41:29.609347 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:29.609351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:29.609637 (+ 286us) server_negotiation.cc:407] Connection header received
0928 02:41:29.609773 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:29.609776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:29.609818 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:29.609877 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:29.610331 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.610894 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.611787 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.612102 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.612176 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.613454 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:29.613458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:29.613461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:29.613591 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:29.615040 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.615428 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.615431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.615432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.615514 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.615766 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.615768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.615770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.615955 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:29.616044 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:29.616302 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:41:29.616412 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:29.632509 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39873
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:29.743795 4495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:29.744102 4495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:29.744169 4495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:29.748435 4495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:29.748508 4495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:29.748567 4495 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:41:29.748592 4495 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:29.748646 4495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:29.754140 4495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:29.755273 4495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:29.756310 4495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:29.765751 4501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:29.765729 4500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:29.765729 4503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:29.765944 4495 server_base.cc:1034] running on GCE node
I20260928 02:41:29.766427 4495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:29.767076 4495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:29.768234 4495 hybrid_clock.cc:648] HybridClock initialized: now 1790563289768213 us; error 32 us; skew 500 ppm
Sep 28 02:41:29 dist-test-slave-rqvb krb5kdc[4238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:29.770833 4495 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:29.771708 4495 webserver.cc:460] Webserver started at http://127.2.123.2:40501/ using document root <none> and password file <none>
I20260928 02:41:29.772261 4495 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:29.772330 4495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:29.772547 4495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:29.774667 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "6cf1afa854654729a173460396cf9a04"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "7ec61d274a86f91ed26c13860cd7fc62"
server_key_iv: "a5a33395266bd5bdc142fc25d707b1bd"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.775198 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "6cf1afa854654729a173460396cf9a04"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "7ec61d274a86f91ed26c13860cd7fc62"
server_key_iv: "a5a33395266bd5bdc142fc25d707b1bd"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.778976 4495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:41:29.781313 4510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:29.782564 4495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:41:29.782672 4495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "6cf1afa854654729a173460396cf9a04"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "7ec61d274a86f91ed26c13860cd7fc62"
server_key_iv: "a5a33395266bd5bdc142fc25d707b1bd"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.782768 4495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:29.799892 4495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:29.802968 4495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:29.803139 4495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:29.804682 4495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:29.804742 4495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:29.804790 4495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:29.804809 4495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:29 dist-test-slave-rqvb krb5kdc[4238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:29.814975 4495 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:33609
I20260928 02:41:29.815048 4625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:33609 every 8 connection(s)
I20260928 02:41:29.815891 4495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:29.817883 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.805519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:48659 (local address 127.2.123.62:39873)
0928 02:41:29.805670 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:29.805675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:29.806643 (+ 968us) server_negotiation.cc:407] Connection header received
0928 02:41:29.807990 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:29.807994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:29.808052 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:29.808160 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:29.809776 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.810460 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.811653 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.811868 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.811953 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.814352 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:29.814355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:29.814357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:29.814381 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:29.815746 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.816447 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.816451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.816453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.816564 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.816943 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.816946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.816947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.817153 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:29.817235 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:29.817654 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 02:41:29.817756 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:41:29.818725 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.805923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39873 (local address 127.2.123.2:48659)
0928 02:41:29.806474 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:29.806536 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:29.807698 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:29.808365 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:29.808385 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:29.808883 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:29.809622 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:29.809644 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.810586 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.810589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:29.811541 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:29.811551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.812025 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.812030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:29.812185 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.812704 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:29.812721 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:29.814192 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:29.815850 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:29.815855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:29.815870 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:29.816294 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:29.816656 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:29.816659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:29.816660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:29.816827 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:29.817237 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:29.817242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:29.817464 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 02:41:29.817611 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:41:29.817861 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":121,"threads_started":1}
I20260928 02:41:29.818938 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 4495
I20260928 02:41:29.819042 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:29.819322 2540 external_mini_cluster.cc:1428] Setting key 54ec370d60acd334f84639ac26fdd648
I20260928 02:41:29.820089 4626 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39873
I20260928 02:41:29.820235 4626 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:29.820784 4626 heartbeater.cc:511] Master 127.2.123.62:39873 requested a full tablet report, sending...
I20260928 02:41:29.821781 4281 ts_manager.cc:194] Registered new tserver with Master: 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609)
I20260928 02:41:29.822476 4281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:48659
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:41:29.829326 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.822194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39873 (local address 127.2.123.2:54153)
0928 02:41:29.822436 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:29.822451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:29.822532 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:29.822801 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:29.822805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:29.822984 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:29.823094 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:29.823101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.824152 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.824156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:29.824722 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:29.824731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.825147 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.825150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:29.825262 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.825718 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:29.825732 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:29.826105 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:29.827812 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:29.827815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:29.827816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:29.828011 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:29.828265 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:29.828267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:29.828268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:29.828332 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:29.828820 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:29.828822 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:29.828916 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 02:41:29.829012 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:41:29.829147 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260928 02:41:29.829361 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:29.822162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:54153 (local address 127.2.123.62:39873)
0928 02:41:29.822485 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:29.822489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:29.822502 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:41:29.822650 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:29.822653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:29.822697 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:29.822770 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:29.823212 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.824027 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.824827 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:29.825053 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:29.825135 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:29.826255 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:29.826260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:29.826263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:29.826290 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:29.827696 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.828103 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.828106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.828108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.828189 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:29.828406 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:29.828409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:29.828411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:29.828703 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:29.828809 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:29.829080 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:41:29.829205 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:29.849078 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39873
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:29.961704 4635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:29.962087 4635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:29.962152 4635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:29.966223 4635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:29.966282 4635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:29.966322 4635 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:41:29.966353 4635 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:29.966403 4635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:29.971659 4635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:29.972728 4635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:29.973753 4635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:29.983467 4643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:29.983474 4640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:29.983536 4635 server_base.cc:1034] running on GCE node
W20260928 02:41:29.983481 4641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:29.984072 4635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:29.984710 4635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:29.985890 4635 hybrid_clock.cc:648] HybridClock initialized: now 1790563289985878 us; error 30 us; skew 500 ppm
Sep 28 02:41:29 dist-test-slave-rqvb krb5kdc[4238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:29.988660 4635 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:29.989533 4635 webserver.cc:460] Webserver started at http://127.2.123.3:43515/ using document root <none> and password file <none>
I20260928 02:41:29.990213 4635 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:29.990298 4635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:29.990535 4635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:29.993012 4635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "f3dee3aca5a04121a5448feb5518c4e3"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "878e3fd557a8ef582fc5179c96205327"
server_key_iv: "eb40f19f5df4af0ab191899ad126dd68"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.993597 4635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "f3dee3aca5a04121a5448feb5518c4e3"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "878e3fd557a8ef582fc5179c96205327"
server_key_iv: "eb40f19f5df4af0ab191899ad126dd68"
server_key_version: "encryptionkey@0"
I20260928 02:41:29.997303 4635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:41:29.999871 4650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:30.001274 4635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:30.001395 4635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "f3dee3aca5a04121a5448feb5518c4e3"
format_stamp: "Formatted at 2026-09-28 02:41:29 on dist-test-slave-rqvb"
server_key: "878e3fd557a8ef582fc5179c96205327"
server_key_iv: "eb40f19f5df4af0ab191899ad126dd68"
server_key_version: "encryptionkey@0"
I20260928 02:41:30.001497 4635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:30.021776 4635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:30.024820 4635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:30.024996 4635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:30.026589 4635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:30.026641 4635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:30.026695 4635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:30.026723 4635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:30 dist-test-slave-rqvb krb5kdc[4238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:30.038605 4635 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:39631
I20260928 02:41:30.038986 4765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:39631 every 8 connection(s)
I20260928 02:41:30.039594 4635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:30.042040 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.027336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:34785 (local address 127.2.123.62:39873)
0928 02:41:30.027460 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.027464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.028556 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 02:41:30.029591 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.029594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.029654 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.029712 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:30.031255 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.031863 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.033623 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.034035 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.034143 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.038330 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:30.038334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:30.038337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:30.038367 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:30.039936 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:30.040530 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:30.040543 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:30.040545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:30.040630 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:30.040945 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:30.040948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:30.040950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:30.041150 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:30.041237 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.041723 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.041854 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:41:30.042836 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.027893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39873 (local address 127.2.123.3:34785)
0928 02:41:30.028387 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.028430 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.029335 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.030159 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.030166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.030499 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:30.031110 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.031127 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.031989 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.031993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.033496 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.033508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.034195 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.034200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.034433 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.035289 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:30.035308 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:30.038086 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:30.040090 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:30.040097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:30.040113 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:30.040421 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:30.040717 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:30.040721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:30.040723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:30.040854 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:30.041247 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:30.041254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:30.041494 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.041664 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.042031 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260928 02:41:30.044128 4766 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39873
I20260928 02:41:30.044253 4766 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:30.044742 4766 heartbeater.cc:511] Master 127.2.123.62:39873 requested a full tablet report, sending...
I20260928 02:41:30.045784 4283 ts_manager.cc:194] Registered new tserver with Master: f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631)
I20260928 02:41:30.045945 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 4635
I20260928 02:41:30.046087 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:30.046379 2540 external_mini_cluster.cc:1428] Setting key ada415ff7d82c57205ef3db6bc0a790d
I20260928 02:41:30.046597 4283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:34785
I20260928 02:41:30.048785 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:30.053220 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.046267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:54273 (local address 127.2.123.62:39873)
0928 02:41:30.046408 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.046413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.046782 (+ 369us) server_negotiation.cc:407] Connection header received
0928 02:41:30.046887 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.046890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.046932 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.047005 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:30.047471 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.048010 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.048950 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.049164 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.049242 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.050416 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:30.050419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:30.050421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:30.050443 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:30.051697 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:30.052152 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:30.052155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:30.052157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:30.052244 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:30.052493 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:30.052495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:30.052497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:30.052654 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:30.052724 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.053012 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.053106 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:41:30.053284 4772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.046396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39873 (local address 127.2.123.3:54273)
0928 02:41:30.046696 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.046711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.046812 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.047022 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.047025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.047240 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:30.047356 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.047363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.048127 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.048131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.048855 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.048865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.049266 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.049270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.049356 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.049836 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:30.049849 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:30.050279 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:30.051801 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:30.051805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:30.051807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:30.052040 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:30.052331 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:30.052334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:30.052336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:30.052421 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:30.052739 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:30.052742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:30.052838 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.052944 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.053095 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
Sep 28 02:41:30 dist-test-slave-rqvb krb5kdc[4238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:41:30 dist-test-slave-rqvb krb5kdc[4238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:30.073539 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.065644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.2.123.62:39873)
0928 02:41:30.065771 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.065774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.065786 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:30.065823 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.065825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.065860 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.065927 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:30.066649 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.067181 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.068137 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.068346 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.068423 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.070755 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:30.070758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:30.070760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:30.070784 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:30.072089 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:30.072457 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:30.072461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:30.072462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:30.072536 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:30.072768 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:30.072771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:30.072773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:30.072940 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:30.073060 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.073279 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.073384 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:41:30.076344 4283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:30.079396 4283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:30.093703 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.088606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.2.123.2:33609)
0928 02:41:30.089348 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.089359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.089380 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:41:30.089441 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.089444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.089640 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.089819 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:30.090063 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.091040 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.092249 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.093065 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.093192 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.093337 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.093502 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.093581 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":105,"threads_started":1}
I20260928 02:41:30.093760 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.088426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33609 (local address 127.0.0.1:44354)
0928 02:41:30.089018 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.089031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.089121 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.089797 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.089801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.089817 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:30.089908 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.089913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.091141 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.091144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.092142 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.092151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.093195 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.093199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.093309 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.093357 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.093482 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.093596 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":23,"thread_start_us":83,"threads_started":1}
I20260928 02:41:30.094589 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.087686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:39631 (local address 127.0.0.1:58498)
0928 02:41:30.088077 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.088129 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.088305 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.089227 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.089231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.089254 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:30.089414 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.089423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.090848 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.090852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.092242 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.092253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.094159 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.094163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.094261 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.094286 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.094378 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.094450 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260928 02:41:30.094828 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.087829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.2.123.3:39631)
0928 02:41:30.088176 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.088186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.088211 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:41:30.088556 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.088564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.088892 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.089055 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:30.089547 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.090728 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.092365 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.093072 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.093172 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.093336 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.094420 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.094514 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":138,"threads_started":1}
I20260928 02:41:30.095208 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.088738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46055 (local address 127.0.0.1:55592)
0928 02:41:30.089640 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.089650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.089730 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.090403 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.090407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.090424 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:30.090509 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.090517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.092335 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.092338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.093088 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.093097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.094384 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.094388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.094904 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.094930 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.095005 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.095087 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":73,"threads_started":1}
I20260928 02:41:30.095683 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.089143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.2.123.1:46055)
0928 02:41:30.089440 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.089456 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.089729 (+ 273us) server_negotiation.cc:407] Connection header received
0928 02:41:30.090122 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.090127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.090263 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.090404 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:30.090628 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.092189 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.093232 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.094045 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.094147 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.094318 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.095433 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.095532 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260928 02:41:30.096851 4560 tablet_service.cc:1467] Processing CreateTablet for tablet b95ec97024f543c79db38fd6b685773e (DEFAULT_TABLE table=test-table [id=1235b53446444557afd95b36ce4644f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:30.097558 4700 tablet_service.cc:1467] Processing CreateTablet for tablet b95ec97024f543c79db38fd6b685773e (DEFAULT_TABLE table=test-table [id=1235b53446444557afd95b36ce4644f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:30.098021 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95ec97024f543c79db38fd6b685773e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.097883 4420 tablet_service.cc:1467] Processing CreateTablet for tablet b95ec97024f543c79db38fd6b685773e (DEFAULT_TABLE table=test-table [id=1235b53446444557afd95b36ce4644f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:30.098596 4700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95ec97024f543c79db38fd6b685773e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.098855 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95ec97024f543c79db38fd6b685773e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.105352 4789 tablet_bootstrap.cc:492] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: Bootstrap starting.
I20260928 02:41:30.106716 4791 tablet_bootstrap.cc:492] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Bootstrap starting.
I20260928 02:41:30.107762 4789 tablet_bootstrap.cc:654] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.108517 4789 log.cc:824] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:30.108685 4791 tablet_bootstrap.cc:654] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.109365 4791 log.cc:824] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:30.109930 4790 tablet_bootstrap.cc:492] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Bootstrap starting.
I20260928 02:41:30.110554 4789 tablet_bootstrap.cc:492] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: No bootstrap required, opened a new log
I20260928 02:41:30.110726 4789 ts_tablet_manager.cc:1397] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:41:30.111690 4790 tablet_bootstrap.cc:654] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.112284 4790 log.cc:824] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:30.113705 4791 tablet_bootstrap.cc:492] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: No bootstrap required, opened a new log
I20260928 02:41:30.113881 4791 ts_tablet_manager.cc:1397] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 02:41:30.113832 4789 raft_consensus.cc:348] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.114149 4789 raft_consensus.cc:374] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.114193 4790 tablet_bootstrap.cc:492] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: No bootstrap required, opened a new log
I20260928 02:41:30.114216 4789 raft_consensus.cc:729] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4129dc7fcb4a07a5e1d7ac055c293f, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.114348 4790 ts_tablet_manager.cc:1397] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 02:41:30.114722 4789 consensus_queue.cc:260] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.115393 4486 heartbeater.cc:503] Master 127.2.123.62:39873 was elected leader, sending a full tablet report...
I20260928 02:41:30.117342 4789 ts_tablet_manager.cc:1428] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:41:30.117306 4791 raft_consensus.cc:348] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.117583 4791 raft_consensus.cc:374] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.117650 4791 raft_consensus.cc:729] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3dee3aca5a04121a5448feb5518c4e3, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.118132 4791 consensus_queue.cc:260] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.118367 4790 raft_consensus.cc:348] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.118614 4790 raft_consensus.cc:374] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.118670 4790 raft_consensus.cc:729] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf1afa854654729a173460396cf9a04, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.119115 4791 ts_tablet_manager.cc:1428] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:41:30.119088 4790 consensus_queue.cc:260] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.119498 4766 heartbeater.cc:503] Master 127.2.123.62:39873 was elected leader, sending a full tablet report...
I20260928 02:41:30.120028 4790 ts_tablet_manager.cc:1428] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:41:30.120358 4626 heartbeater.cc:503] Master 127.2.123.62:39873 was elected leader, sending a full tablet report...
I20260928 02:41:30.246920 4796 raft_consensus.cc:482] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:30.247128 4796 raft_consensus.cc:504] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.248281 4796 leader_election.cc:288] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609), 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055)
I20260928 02:41:30.253224 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.248706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:32955 (local address 127.2.123.2:33609)
0928 02:41:30.248833 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.248838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.248854 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:41:30.248907 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.248910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.248955 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.249020 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:30.249358 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.250144 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.252034 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.252637 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.252741 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.252804 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.253024 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.253095 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:41:30.253242 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.248982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:39759 (local address 127.2.123.1:46055)
0928 02:41:30.249129 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.249133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.249228 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:41:30.249332 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.249335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.249382 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.249457 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:30.249679 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.250382 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.252031 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.252636 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.252741 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.252804 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.253081 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.253145 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:41:30.253268 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.248827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46055 (local address 127.2.123.3:39759)
0928 02:41:30.249143 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.249155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.249246 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.249475 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.249477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.249492 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:30.249584 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.249588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.250485 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.250489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.251921 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.251937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.252754 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.252758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.252908 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.252980 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.253042 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.253095 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":70,"thread_start_us":117,"threads_started":1}
I20260928 02:41:30.253268 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.248555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33609 (local address 127.2.123.3:32955)
0928 02:41:30.248694 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.248708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.248808 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.249112 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.249115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.249138 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:30.249258 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.249265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.250241 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.250243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.251921 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.251933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.252754 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.252758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.252895 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.252933 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.253012 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.253095 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":70}
I20260928 02:41:30.254021 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b95ec97024f543c79db38fd6b685773e" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" is_pre_election: true
I20260928 02:41:30.254016 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b95ec97024f543c79db38fd6b685773e" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1afa854654729a173460396cf9a04" is_pre_election: true
I20260928 02:41:30.254355 4440 raft_consensus.cc:2454] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 0.
I20260928 02:41:30.254355 4580 raft_consensus.cc:2454] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 0.
I20260928 02:41:30.254827 4654 leader_election.cc:302] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4129dc7fcb4a07a5e1d7ac055c293f, f3dee3aca5a04121a5448feb5518c4e3; no voters:
I20260928 02:41:30.255105 4796 raft_consensus.cc:2781] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:30.255210 4796 raft_consensus.cc:482] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:30.255254 4796 raft_consensus.cc:3037] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.256296 4796 raft_consensus.cc:504] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.256599 4796 leader_election.cc:288] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 election: Requested vote from peers 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609), 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055)
I20260928 02:41:30.257064 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b95ec97024f543c79db38fd6b685773e" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f"
I20260928 02:41:30.257063 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b95ec97024f543c79db38fd6b685773e" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1afa854654729a173460396cf9a04"
I20260928 02:41:30.257196 4580 raft_consensus.cc:3037] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.257197 4440 raft_consensus.cc:3037] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.258476 4580 raft_consensus.cc:2454] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 1.
I20260928 02:41:30.258492 4440 raft_consensus.cc:2454] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 1.
I20260928 02:41:30.258778 4654 leader_election.cc:302] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf1afa854654729a173460396cf9a04, f3dee3aca5a04121a5448feb5518c4e3; no voters:
I20260928 02:41:30.258970 4796 raft_consensus.cc:2781] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:30.259183 4796 raft_consensus.cc:686] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 1 LEADER]: Becoming Leader. State: Replica: f3dee3aca5a04121a5448feb5518c4e3, State: Running, Role: LEADER
I20260928 02:41:30.259527 4796 consensus_queue.cc:237] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } }
I20260928 02:41:30.262701 4281 catalog_manager.cc:5495] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3). New cstate: current_term: 1 leader_uuid: "f3dee3aca5a04121a5448feb5518c4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } health_report { overall_health: UNKNOWN } } }
W20260928 02:41:30.290549 4767 tablet.cc:2369] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:41:30.317174 4627 tablet.cc:2369] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:41:30 dist-test-slave-rqvb krb5kdc[4238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:30.324082 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.315741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39873 (local address 127.0.0.1:48662)
0928 02:41:30.316154 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.316172 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.316308 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.316551 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.316553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.316950 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:30.317075 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.317083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.317886 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.317889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.318485 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.318494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.318894 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.318896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.319015 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.319513 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:30.319523 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:30.321018 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:30.322660 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:30.322669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:30.322676 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:30.322894 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:30.323129 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:30.323137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:30.323139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:30.323207 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:30.323542 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:30.323549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:30.323653 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.323745 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.323899 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":111,"threads_started":1}
I20260928 02:41:30.324156 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.315825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.2.123.62:39873)
0928 02:41:30.316013 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.316018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.316261 (+ 243us) server_negotiation.cc:407] Connection header received
0928 02:41:30.316408 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.316410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.316450 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.316518 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:30.317211 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.317783 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.318592 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.318803 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.318891 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.321153 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:30.321157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:30.321159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:30.321180 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:30.322567 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:30.322985 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:30.322989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:30.322990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:30.323055 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:30.323282 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:30.323285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:30.323287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:30.323451 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:30.323532 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.323800 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.324017 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":80}
I20260928 02:41:30.328094 4281 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:48662:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:41:30.328816 4281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:30.333575 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 746f316bcf334417afb46e52e0263474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f9a0cd42c0b4881a97925c797e1a57a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:30.334331 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746f316bcf334417afb46e52e0263474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.334314 4700 tablet_service.cc:1467] Processing CreateTablet for tablet 746f316bcf334417afb46e52e0263474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f9a0cd42c0b4881a97925c797e1a57a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:30.334860 4700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746f316bcf334417afb46e52e0263474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.337049 4791 tablet_bootstrap.cc:492] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: Bootstrap starting.
I20260928 02:41:30.337087 4790 tablet_bootstrap.cc:492] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: Bootstrap starting.
I20260928 02:41:30.333714 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 746f316bcf334417afb46e52e0263474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f9a0cd42c0b4881a97925c797e1a57a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:30.337723 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746f316bcf334417afb46e52e0263474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.337919 4790 tablet_bootstrap.cc:654] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.338035 4791 tablet_bootstrap.cc:654] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.339352 4791 tablet_bootstrap.cc:492] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: No bootstrap required, opened a new log
I20260928 02:41:30.339428 4791 ts_tablet_manager.cc:1397] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:30.339639 4790 tablet_bootstrap.cc:492] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: No bootstrap required, opened a new log
I20260928 02:41:30.339725 4790 ts_tablet_manager.cc:1397] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:41:30.339823 4791 raft_consensus.cc:348] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.339919 4791 raft_consensus.cc:374] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.339946 4791 raft_consensus.cc:729] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3dee3aca5a04121a5448feb5518c4e3, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.340070 4791 consensus_queue.cc:260] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.340145 4790 raft_consensus.cc:348] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.340243 4790 raft_consensus.cc:374] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.340270 4790 raft_consensus.cc:729] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf1afa854654729a173460396cf9a04, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.340304 4791 ts_tablet_manager.cc:1428] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:30.340379 4790 consensus_queue.cc:260] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.340502 4796 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.340585 4789 tablet_bootstrap.cc:492] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Bootstrap starting.
I20260928 02:41:30.340631 4796 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:30.340669 4790 ts_tablet_manager.cc:1428] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:30.340950 4797 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.341082 4797 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:30.341528 4789 tablet_bootstrap.cc:654] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.347570 4789 tablet_bootstrap.cc:492] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: No bootstrap required, opened a new log
I20260928 02:41:30.347678 4789 ts_tablet_manager.cc:1397] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 02:41:30.348061 4789 raft_consensus.cc:348] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.348155 4789 raft_consensus.cc:374] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.348182 4789 raft_consensus.cc:729] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4129dc7fcb4a07a5e1d7ac055c293f, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.348296 4789 consensus_queue.cc:260] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.348572 4789 ts_tablet_manager.cc:1428] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:30.348616 4795 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.348769 4795 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:41:30.354050 4487 tablet.cc:2369] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:30.479607 4797 raft_consensus.cc:482] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:30.479758 4797 raft_consensus.cc:504] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.480887 4797 leader_election.cc:288] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055), f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631)
I20260928 02:41:30.484833 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.481702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:46365 (local address 127.2.123.3:39631)
0928 02:41:30.481817 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.481821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.481901 (+ 80us) server_negotiation.cc:407] Connection header received
0928 02:41:30.482108 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.482111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.482153 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.482215 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:30.482472 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.482951 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.483922 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.484307 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.484397 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.484468 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.484648 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.484715 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:41:30.484858 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.481605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:39631 (local address 127.2.123.2:46365)
0928 02:41:30.481827 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.481839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.481945 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.482233 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.482236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.482252 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:30.482370 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.482375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.483053 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.483056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.483810 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.483818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.484431 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.484435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.484540 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.484562 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.484623 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.484688 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260928 02:41:30.484899 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.481283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:54103 (local address 127.2.123.1:46055)
0928 02:41:30.481406 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.481410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.481557 (+ 147us) server_negotiation.cc:407] Connection header received
0928 02:41:30.481775 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.481779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.481847 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.481897 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:30.482296 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.482807 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.483961 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.484369 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.484472 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.484537 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.484705 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.484774 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:41:30.484922 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.481192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46055 (local address 127.2.123.2:54103)
0928 02:41:30.481447 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.481463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.481593 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.482066 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.482068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.482085 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:30.482187 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.482193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.482964 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.482966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.483821 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.483830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.484485 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.484489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.484584 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.484603 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.484668 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.484740 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260928 02:41:30.485589 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746f316bcf334417afb46e52e0263474" candidate_uuid: "6cf1afa854654729a173460396cf9a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" is_pre_election: true
I20260928 02:41:30.485600 4720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746f316bcf334417afb46e52e0263474" candidate_uuid: "6cf1afa854654729a173460396cf9a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3dee3aca5a04121a5448feb5518c4e3" is_pre_election: true
I20260928 02:41:30.485769 4440 raft_consensus.cc:2454] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf1afa854654729a173460396cf9a04 in term 0.
I20260928 02:41:30.485826 4720 raft_consensus.cc:2454] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf1afa854654729a173460396cf9a04 in term 0.
I20260928 02:41:30.486176 4514 leader_election.cc:302] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4129dc7fcb4a07a5e1d7ac055c293f, 6cf1afa854654729a173460396cf9a04; no voters:
I20260928 02:41:30.486479 4797 raft_consensus.cc:2781] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:30.486542 4797 raft_consensus.cc:482] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:30.486586 4797 raft_consensus.cc:3037] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.487654 4797 raft_consensus.cc:504] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.488047 4797 leader_election.cc:288] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055), f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631)
I20260928 02:41:30.488447 4720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746f316bcf334417afb46e52e0263474" candidate_uuid: "6cf1afa854654729a173460396cf9a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3dee3aca5a04121a5448feb5518c4e3"
I20260928 02:41:30.488442 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746f316bcf334417afb46e52e0263474" candidate_uuid: "6cf1afa854654729a173460396cf9a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f"
I20260928 02:41:30.488569 4720 raft_consensus.cc:3037] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.488569 4440 raft_consensus.cc:3037] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.489740 4440 raft_consensus.cc:2454] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf1afa854654729a173460396cf9a04 in term 1.
I20260928 02:41:30.489734 4720 raft_consensus.cc:2454] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf1afa854654729a173460396cf9a04 in term 1.
I20260928 02:41:30.490022 4514 leader_election.cc:302] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4129dc7fcb4a07a5e1d7ac055c293f, 6cf1afa854654729a173460396cf9a04; no voters:
I20260928 02:41:30.490250 4797 raft_consensus.cc:2781] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:30.490447 4797 raft_consensus.cc:686] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf1afa854654729a173460396cf9a04, State: Running, Role: LEADER
I20260928 02:41:30.490787 4797 consensus_queue.cc:237] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.492805 4812 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: TxnStatusTablet state changed. Reason: New leader 6cf1afa854654729a173460396cf9a04. Latest consensus state: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.492918 4812 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:30.493286 4814 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:41:30.493507 4814 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:41:30.493820 4281 catalog_manager.cc:5495] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cf1afa854654729a173460396cf9a04 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:30.535002 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.531202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33609 (local address 127.0.0.1:44360)
0928 02:41:30.531350 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.531366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.531470 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.531777 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.531780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.531788 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:30.531891 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.531896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.532588 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.532591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.533187 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.533194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.533674 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.533679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.533785 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.533839 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.534670 (+ 831us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.534684 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.534757 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.534847 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:41:30.535034 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.531321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.2.123.2:33609)
0928 02:41:30.531457 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.531461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.531476 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:41:30.531616 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.531619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.531671 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.531749 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:30.531983 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.532487 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.533287 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.533547 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.533642 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.534129 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.534575 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.534668 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.534803 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.534917 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:41:30.540136 4440 raft_consensus.cc:1261] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Refusing update from remote peer 6cf1afa854654729a173460396cf9a04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:30.540161 4720 raft_consensus.cc:1261] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Refusing update from remote peer 6cf1afa854654729a173460396cf9a04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:30.540743 4797 consensus_queue.cc:1035] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:30.540934 4812 consensus_queue.cc:1035] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:30.544123 4796 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: TxnStatusTablet state changed. Reason: New leader 6cf1afa854654729a173460396cf9a04. Latest consensus state: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.544245 4796 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:30.545095 4795 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: TxnStatusTablet state changed. Reason: New leader 6cf1afa854654729a173460396cf9a04. Latest consensus state: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.545221 4795 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:30.548211 4797 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.548333 4797 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:30.548451 4817 mvcc.cc:205] Tried to move back new op lower bound from 7334147238045880320 to 7334147237852725248. Current Snapshot: MvccSnapshot[applied={T|T < 7334147238045880320}]
I20260928 02:41:30.548566 4795 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.548681 4795 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P 3c4129dc7fcb4a07a5e1d7ac055c293f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:30.548921 4818 mvcc.cc:205] Tried to move back new op lower bound from 7334147238045880320 to 7334147237852725248. Current Snapshot: MvccSnapshot[applied={T|T < 7334147238045880320}]
I20260928 02:41:30.551074 4812 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.551178 4812 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P 6cf1afa854654729a173460396cf9a04: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:30.552229 4796 tablet_replica.cc:439] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } } }
I20260928 02:41:30.552336 4796 tablet_replica.cc:442] T 746f316bcf334417afb46e52e0263474 P f3dee3aca5a04121a5448feb5518c4e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:30.559669 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.555890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.2.123.3:39631)
0928 02:41:30.556001 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.556004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.556052 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:41:30.556219 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.556221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.556267 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.556360 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:30.556591 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.557074 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.558278 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.558489 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.558565 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.558912 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.559289 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.559391 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.559475 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.559524 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:41:30.570683 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.567059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:53807 (local address 127.2.123.2:33609)
0928 02:41:30.567171 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.567174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.567277 (+ 103us) server_negotiation.cc:407] Connection header received
0928 02:41:30.567399 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.567401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.567445 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.567514 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:30.567738 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.568397 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.569625 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.569862 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.569948 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.570311 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.570381 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.570451 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.570537 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.570579 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:41:30.570756 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.566880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33609 (local address 127.2.123.3:53807)
0928 02:41:30.567211 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.567225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.567322 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.567529 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.567540 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.567547 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:30.567654 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.567659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.568499 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.568502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.569489 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.569499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.569996 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.570000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.570092 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.570141 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.570461 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.570467 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.570530 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.570611 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
I20260928 02:41:30.577651 4440 raft_consensus.cc:1261] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Refusing update from remote peer f3dee3aca5a04121a5448feb5518c4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:30.577832 4580 raft_consensus.cc:1261] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Refusing update from remote peer f3dee3aca5a04121a5448feb5518c4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:30.578138 4799 consensus_queue.cc:1035] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:30.578356 4796 consensus_queue.cc:1035] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:30.585363 4819 mvcc.cc:205] Tried to move back new op lower bound from 7334147238201966592 to 7334147236905635840. Current Snapshot: MvccSnapshot[applied={T|T < 7334147238201966592}]
I20260928 02:41:30.602341 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.597153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:50699 (local address 127.2.123.3:39631)
0928 02:41:30.597267 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.597272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.597632 (+ 360us) server_negotiation.cc:407] Connection header received
0928 02:41:30.597741 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.597745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.597794 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.597859 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:30.598203 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.598726 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.599987 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.600320 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.600403 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.600869 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.600959 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.601031 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.602142 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.602202 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:41:30.602362 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.597075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:39631 (local address 127.2.123.2:50699)
0928 02:41:30.597554 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.597569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.597663 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.597874 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.597877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.597887 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:30.598058 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.598065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.598846 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.598850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.599820 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.599832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.600576 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.600580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.600689 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.600743 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.602021 (+ 1278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.602030 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.602123 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.602210 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":91,"threads_started":1}
I20260928 02:41:30.628093 4283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:30.628304 4283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:30.631237 4283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: Sending DeleteTablet for 3 replicas of tablet b95ec97024f543c79db38fd6b685773e
I20260928 02:41:30.632130 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet b95ec97024f543c79db38fd6b685773e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:30 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:55592
I20260928 02:41:30.632318 4700 tablet_service.cc:1514] Processing DeleteTablet for tablet b95ec97024f543c79db38fd6b685773e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:30 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:58498
I20260928 02:41:30.632802 4838 tablet_replica.cc:330] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: stopping tablet replica
I20260928 02:41:30.632766 4560 tablet_service.cc:1514] Processing DeleteTablet for tablet b95ec97024f543c79db38fd6b685773e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:30 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:44354
I20260928 02:41:30.633123 4838 raft_consensus.cc:2229] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:30.633489 4838 raft_consensus.cc:2258] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:30.634287 4283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:30.634708 4283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:30.634869 4838 ts_tablet_manager.cc:1905] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:30.637888 4838 ts_tablet_manager.cc:1918] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:30.638039 4838 log.cc:1196] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/b95ec97024f543c79db38fd6b685773e
I20260928 02:41:30.638321 4838 ts_tablet_manager.cc:1939] T b95ec97024f543c79db38fd6b685773e P f3dee3aca5a04121a5448feb5518c4e3: Deleting consensus metadata
I20260928 02:41:30.639451 4270 catalog_manager.cc:4841] TS f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631): tablet b95ec97024f543c79db38fd6b685773e (table test-table [id=1235b53446444557afd95b36ce4644f0]) successfully deleted
I20260928 02:41:30.640558 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 118e8d4507e44bb989ea55b9854066e0 (DEFAULT_TABLE table=test-table [id=be41c23239884031b6ff6afdc3db83ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:30.640856 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118e8d4507e44bb989ea55b9854066e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.641033 4839 tablet_replica.cc:330] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: stopping tablet replica
I20260928 02:41:30.641407 4839 raft_consensus.cc:2229] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:30.641647 4839 raft_consensus.cc:2258] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:30.643469 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 118e8d4507e44bb989ea55b9854066e0 (DEFAULT_TABLE table=test-table [id=be41c23239884031b6ff6afdc3db83ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:30.643728 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118e8d4507e44bb989ea55b9854066e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.644006 4839 ts_tablet_manager.cc:1905] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:30.646109 4790 tablet_bootstrap.cc:492] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: Bootstrap starting.
I20260928 02:41:30.646966 4790 tablet_bootstrap.cc:654] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.647783 4700 tablet_service.cc:1467] Processing CreateTablet for tablet 118e8d4507e44bb989ea55b9854066e0 (DEFAULT_TABLE table=test-table [id=be41c23239884031b6ff6afdc3db83ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:30.648056 4700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118e8d4507e44bb989ea55b9854066e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:30.648830 4839 ts_tablet_manager.cc:1918] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:30.648983 4839 log.cc:1196] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/b95ec97024f543c79db38fd6b685773e
I20260928 02:41:30.649353 4839 ts_tablet_manager.cc:1939] T b95ec97024f543c79db38fd6b685773e P 6cf1afa854654729a173460396cf9a04: Deleting consensus metadata
I20260928 02:41:30.650192 4789 tablet_bootstrap.cc:492] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Bootstrap starting.
I20260928 02:41:30.650281 4791 tablet_bootstrap.cc:492] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: Bootstrap starting.
I20260928 02:41:30.650370 4837 tablet_replica.cc:330] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: stopping tablet replica
I20260928 02:41:30.651106 4789 tablet_bootstrap.cc:654] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.651140 4791 tablet_bootstrap.cc:654] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:30.651561 4837 raft_consensus.cc:2229] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:30.652290 4270 catalog_manager.cc:4841] TS 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609): tablet b95ec97024f543c79db38fd6b685773e (table test-table [id=1235b53446444557afd95b36ce4644f0]) successfully deleted
I20260928 02:41:30.652652 4837 raft_consensus.cc:2258] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:30.652668 4791 tablet_bootstrap.cc:492] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: No bootstrap required, opened a new log
I20260928 02:41:30.652740 4791 ts_tablet_manager.cc:1397] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:30.653189 4791 raft_consensus.cc:348] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.653311 4791 raft_consensus.cc:374] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.653349 4791 raft_consensus.cc:729] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3dee3aca5a04121a5448feb5518c4e3, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.653479 4791 consensus_queue.cc:260] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.653730 4791 ts_tablet_manager.cc:1428] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:30.654258 4790 tablet_bootstrap.cc:492] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: No bootstrap required, opened a new log
I20260928 02:41:30.654268 4837 ts_tablet_manager.cc:1905] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:30.654343 4790 ts_tablet_manager.cc:1397] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 02:41:30.654757 4790 raft_consensus.cc:348] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.654867 4790 raft_consensus.cc:374] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.654897 4790 raft_consensus.cc:729] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf1afa854654729a173460396cf9a04, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.655014 4790 consensus_queue.cc:260] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.655246 4790 ts_tablet_manager.cc:1428] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:30.659927 4837 ts_tablet_manager.cc:1918] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:30.660091 4837 log.cc:1196] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/b95ec97024f543c79db38fd6b685773e
I20260928 02:41:30.660439 4837 ts_tablet_manager.cc:1939] T b95ec97024f543c79db38fd6b685773e P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting consensus metadata
I20260928 02:41:30.661712 4270 catalog_manager.cc:4841] TS 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055): tablet b95ec97024f543c79db38fd6b685773e (table test-table [id=1235b53446444557afd95b36ce4644f0]) successfully deleted
I20260928 02:41:30.662726 4789 tablet_bootstrap.cc:492] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: No bootstrap required, opened a new log
I20260928 02:41:30.662819 4789 ts_tablet_manager.cc:1397] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 02:41:30.663262 4789 raft_consensus.cc:348] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.663383 4789 raft_consensus.cc:374] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:30.663411 4789 raft_consensus.cc:729] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4129dc7fcb4a07a5e1d7ac055c293f, State: Initialized, Role: FOLLOWER
I20260928 02:41:30.663537 4789 consensus_queue.cc:260] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.663795 4789 ts_tablet_manager.cc:1428] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:30.802083 4797 raft_consensus.cc:482] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:30.802232 4797 raft_consensus.cc:504] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.802621 4797 leader_election.cc:288] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055), f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631)
I20260928 02:41:30.803097 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118e8d4507e44bb989ea55b9854066e0" candidate_uuid: "6cf1afa854654729a173460396cf9a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" is_pre_election: true
I20260928 02:41:30.803135 4720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118e8d4507e44bb989ea55b9854066e0" candidate_uuid: "6cf1afa854654729a173460396cf9a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3dee3aca5a04121a5448feb5518c4e3" is_pre_election: true
I20260928 02:41:30.803258 4720 raft_consensus.cc:2454] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf1afa854654729a173460396cf9a04 in term 0.
I20260928 02:41:30.803258 4440 raft_consensus.cc:2454] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf1afa854654729a173460396cf9a04 in term 0.
I20260928 02:41:30.803622 4514 leader_election.cc:302] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf1afa854654729a173460396cf9a04, f3dee3aca5a04121a5448feb5518c4e3; no voters:
I20260928 02:41:30.803814 4797 raft_consensus.cc:2781] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:30.803879 4797 raft_consensus.cc:482] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:30.804047 4797 raft_consensus.cc:3037] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.805140 4797 raft_consensus.cc:504] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.805476 4797 leader_election.cc:288] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055), f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631)
I20260928 02:41:30.805898 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118e8d4507e44bb989ea55b9854066e0" candidate_uuid: "6cf1afa854654729a173460396cf9a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f"
I20260928 02:41:30.805898 4720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118e8d4507e44bb989ea55b9854066e0" candidate_uuid: "6cf1afa854654729a173460396cf9a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3dee3aca5a04121a5448feb5518c4e3"
I20260928 02:41:30.806069 4720 raft_consensus.cc:3037] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.806068 4440 raft_consensus.cc:3037] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:30.807258 4440 raft_consensus.cc:2454] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf1afa854654729a173460396cf9a04 in term 1.
I20260928 02:41:30.807258 4720 raft_consensus.cc:2454] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf1afa854654729a173460396cf9a04 in term 1.
I20260928 02:41:30.807601 4514 leader_election.cc:302] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf1afa854654729a173460396cf9a04, f3dee3aca5a04121a5448feb5518c4e3; no voters:
I20260928 02:41:30.807776 4797 raft_consensus.cc:2781] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:30.807873 4797 raft_consensus.cc:686] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf1afa854654729a173460396cf9a04, State: Running, Role: LEADER
I20260928 02:41:30.808022 4797 consensus_queue.cc:237] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:30.809584 4283 catalog_manager.cc:5495] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cf1afa854654729a173460396cf9a04 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "6cf1afa854654729a173460396cf9a04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:30.843748 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.840698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.2.123.2:33609)
0928 02:41:30.840804 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.840807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.840819 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:30.840902 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.840904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.840943 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.841005 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:30.841225 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.841722 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.842765 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.842980 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.843068 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.843296 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.843362 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.843437 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.843559 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.843610 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:41:30.852056 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.849363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:57199 (local address 127.2.123.2:33609)
0928 02:41:30.849462 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:30.849466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:30.849476 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:41:30.849595 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:30.849598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:30.849637 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:30.849696 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:30.849919 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.850477 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.851255 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.851450 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.851518 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.851725 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.851785 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.851830 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:30.851888 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 02:41:30.851923 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:41:30.852087 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:30.849244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33609 (local address 127.2.123.2:57199)
0928 02:41:30.849397 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:30.849412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:30.849486 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:30.849741 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:30.849743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:30.849750 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:30.849835 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.849840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.850587 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.850590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.851166 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:30.851173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:30.851558 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:30.851561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:30.851617 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:30.851642 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:30.851837 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:30.851841 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:41:30.851882 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 02:41:30.851947 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:41:30.857678 4720 raft_consensus.cc:1261] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Refusing update from remote peer 6cf1afa854654729a173460396cf9a04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:30.857702 4440 raft_consensus.cc:1261] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Refusing update from remote peer 6cf1afa854654729a173460396cf9a04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:30.858031 4812 consensus_queue.cc:1035] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:30.858183 4797 consensus_queue.cc:1035] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:41:30.871757 4817 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:41:32.425243 4281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:32.425441 4281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:32.427875 4281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: Sending DeleteTablet for 3 replicas of tablet 118e8d4507e44bb989ea55b9854066e0
I20260928 02:41:32.428539 4560 tablet_service.cc:1514] Processing DeleteTablet for tablet 118e8d4507e44bb989ea55b9854066e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:32 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:44354
I20260928 02:41:32.428522 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet 118e8d4507e44bb989ea55b9854066e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:32 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:55592
I20260928 02:41:32.428617 4700 tablet_service.cc:1514] Processing DeleteTablet for tablet 118e8d4507e44bb989ea55b9854066e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:32 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:58498
I20260928 02:41:32.428982 4869 tablet_replica.cc:330] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: stopping tablet replica
I20260928 02:41:32.429124 4869 raft_consensus.cc:2229] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:32.429266 4869 raft_consensus.cc:2258] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:32.429922 4869 ts_tablet_manager.cc:1905] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:32.432122 4869 ts_tablet_manager.cc:1918] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:41:32.432188 4869 log.cc:1196] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/118e8d4507e44bb989ea55b9854066e0
I20260928 02:41:32.432425 4869 ts_tablet_manager.cc:1939] T 118e8d4507e44bb989ea55b9854066e0 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting consensus metadata
I20260928 02:41:32.432466 4868 tablet_replica.cc:330] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: stopping tablet replica
I20260928 02:41:32.432595 4868 raft_consensus.cc:2229] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:32.432953 4868 raft_consensus.cc:2258] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:32.433207 4870 tablet_replica.cc:330] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: stopping tablet replica
I20260928 02:41:32.433324 4270 catalog_manager.cc:4841] TS 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055): tablet 118e8d4507e44bb989ea55b9854066e0 (table test-table [id=be41c23239884031b6ff6afdc3db83ff]) successfully deleted
I20260928 02:41:32.433336 4870 raft_consensus.cc:2229] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:32.433748 4870 raft_consensus.cc:2258] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:32.434255 4868 ts_tablet_manager.cc:1905] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:32.434787 4870 ts_tablet_manager.cc:1905] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:32.436637 4868 ts_tablet_manager.cc:1918] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:41:32.436705 4868 log.cc:1196] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/118e8d4507e44bb989ea55b9854066e0
I20260928 02:41:32.436952 4868 ts_tablet_manager.cc:1939] T 118e8d4507e44bb989ea55b9854066e0 P 6cf1afa854654729a173460396cf9a04: Deleting consensus metadata
I20260928 02:41:32.437233 4870 ts_tablet_manager.cc:1918] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:41:32.437301 4870 log.cc:1196] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/118e8d4507e44bb989ea55b9854066e0
I20260928 02:41:32.437515 4870 ts_tablet_manager.cc:1939] T 118e8d4507e44bb989ea55b9854066e0 P f3dee3aca5a04121a5448feb5518c4e3: Deleting consensus metadata
I20260928 02:41:32.437623 4270 catalog_manager.cc:4841] TS 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609): tablet 118e8d4507e44bb989ea55b9854066e0 (table test-table [id=be41c23239884031b6ff6afdc3db83ff]) successfully deleted
I20260928 02:41:32.438318 4270 catalog_manager.cc:4841] TS f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631): tablet 118e8d4507e44bb989ea55b9854066e0 (table test-table [id=be41c23239884031b6ff6afdc3db83ff]) successfully deleted
Sep 28 02:41:32 dist-test-slave-rqvb krb5kdc[4238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:41:32 dist-test-slave-rqvb krb5kdc[4238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:32.462958 4878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:32.448332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.2.123.62:39873)
0928 02:41:32.448615 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:32.448621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:32.448637 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:41:32.448744 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:32.448746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:32.448791 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:32.448876 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:32.449492 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:32.450139 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:32.451045 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:32.451320 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:32.451408 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:32.459958 (+ 8550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:32.459962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:32.459965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:32.459997 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:32.461369 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:32.461764 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:32.461766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:32.461768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:32.461853 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:32.462135 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:32.462138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:32.462140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:32.462348 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:32.462455 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:32.462706 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:41:32.462826 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
I20260928 02:41:32.465816 4281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:32.466331 4281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:32.471026 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 62756b8d0efd4fa39bb1a84e8958c457 (DEFAULT_TABLE table=test-table [id=7b8124063d3446c4bc31e93d0991fa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:32.471098 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 62756b8d0efd4fa39bb1a84e8958c457 (DEFAULT_TABLE table=test-table [id=7b8124063d3446c4bc31e93d0991fa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:32.471217 4700 tablet_service.cc:1467] Processing CreateTablet for tablet 62756b8d0efd4fa39bb1a84e8958c457 (DEFAULT_TABLE table=test-table [id=7b8124063d3446c4bc31e93d0991fa9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:32.471359 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62756b8d0efd4fa39bb1a84e8958c457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:32.471407 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62756b8d0efd4fa39bb1a84e8958c457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:32.471500 4700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62756b8d0efd4fa39bb1a84e8958c457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:32.473587 4879 tablet_bootstrap.cc:492] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: Bootstrap starting.
I20260928 02:41:32.474154 4880 tablet_bootstrap.cc:492] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: Bootstrap starting.
I20260928 02:41:32.474474 4879 tablet_bootstrap.cc:654] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:32.474992 4880 tablet_bootstrap.cc:654] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:32.475489 4879 tablet_bootstrap.cc:492] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: No bootstrap required, opened a new log
I20260928 02:41:32.475538 4881 tablet_bootstrap.cc:492] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Bootstrap starting.
I20260928 02:41:32.475627 4879 ts_tablet_manager.cc:1397] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:32.476001 4879 raft_consensus.cc:348] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.476083 4880 tablet_bootstrap.cc:492] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: No bootstrap required, opened a new log
I20260928 02:41:32.476099 4879 raft_consensus.cc:374] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:32.476128 4879 raft_consensus.cc:729] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf1afa854654729a173460396cf9a04, State: Initialized, Role: FOLLOWER
I20260928 02:41:32.476158 4880 ts_tablet_manager.cc:1397] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:32.476277 4879 consensus_queue.cc:260] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.476485 4881 tablet_bootstrap.cc:654] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:32.476594 4880 raft_consensus.cc:348] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.476712 4880 raft_consensus.cc:374] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:32.476751 4880 raft_consensus.cc:729] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3dee3aca5a04121a5448feb5518c4e3, State: Initialized, Role: FOLLOWER
I20260928 02:41:32.476897 4880 consensus_queue.cc:260] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.477084 4879 ts_tablet_manager.cc:1428] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:32.478205 4881 tablet_bootstrap.cc:492] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: No bootstrap required, opened a new log
I20260928 02:41:32.478295 4881 ts_tablet_manager.cc:1397] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:32.478293 4880 ts_tablet_manager.cc:1428] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:32.478849 4881 raft_consensus.cc:348] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.478961 4881 raft_consensus.cc:374] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:32.478991 4881 raft_consensus.cc:729] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4129dc7fcb4a07a5e1d7ac055c293f, State: Initialized, Role: FOLLOWER
I20260928 02:41:32.479127 4881 consensus_queue.cc:260] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.479498 4881 ts_tablet_manager.cc:1428] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:32.736059 4885 raft_consensus.cc:482] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:32.736212 4885 raft_consensus.cc:504] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.736629 4885 leader_election.cc:288] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055), 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609)
I20260928 02:41:32.737121 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62756b8d0efd4fa39bb1a84e8958c457" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" is_pre_election: true
I20260928 02:41:32.737191 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62756b8d0efd4fa39bb1a84e8958c457" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1afa854654729a173460396cf9a04" is_pre_election: true
I20260928 02:41:32.737329 4580 raft_consensus.cc:2454] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 0.
I20260928 02:41:32.737342 4440 raft_consensus.cc:2454] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 0.
I20260928 02:41:32.737684 4654 leader_election.cc:302] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4129dc7fcb4a07a5e1d7ac055c293f, f3dee3aca5a04121a5448feb5518c4e3; no voters:
I20260928 02:41:32.737896 4885 raft_consensus.cc:2781] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:32.737993 4885 raft_consensus.cc:482] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:32.738046 4885 raft_consensus.cc:3037] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:32.739117 4885 raft_consensus.cc:504] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.739460 4885 leader_election.cc:288] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055), 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609)
I20260928 02:41:32.739869 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62756b8d0efd4fa39bb1a84e8958c457" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f"
I20260928 02:41:32.739872 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62756b8d0efd4fa39bb1a84e8958c457" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1afa854654729a173460396cf9a04"
I20260928 02:41:32.739995 4580 raft_consensus.cc:3037] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:32.739995 4440 raft_consensus.cc:3037] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:32.741086 4440 raft_consensus.cc:2454] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 1.
I20260928 02:41:32.741086 4580 raft_consensus.cc:2454] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 1.
I20260928 02:41:32.741372 4654 leader_election.cc:302] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf1afa854654729a173460396cf9a04, f3dee3aca5a04121a5448feb5518c4e3; no voters:
I20260928 02:41:32.741528 4885 raft_consensus.cc:2781] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:32.741755 4885 raft_consensus.cc:686] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 LEADER]: Becoming Leader. State: Replica: f3dee3aca5a04121a5448feb5518c4e3, State: Running, Role: LEADER
I20260928 02:41:32.741881 4885 consensus_queue.cc:237] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.743588 4283 catalog_manager.cc:5495] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3). New cstate: current_term: 1 leader_uuid: "f3dee3aca5a04121a5448feb5518c4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } health_report { overall_health: HEALTHY } } }
I20260928 02:41:32.777953 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:32.774810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.2.123.3:39631)
0928 02:41:32.775076 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:32.775082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:32.775096 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:41:32.775138 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:32.775141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:32.775180 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:32.775290 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:32.775540 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:32.776049 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:32.776868 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:32.777122 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:32.777285 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:32.777560 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:32.777633 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:32.777696 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:32.777764 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:41:32.777809 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":133,"threads_started":1}
I20260928 02:41:32.783789 4580 raft_consensus.cc:1261] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Refusing update from remote peer f3dee3aca5a04121a5448feb5518c4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:32.783789 4440 raft_consensus.cc:1261] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Refusing update from remote peer f3dee3aca5a04121a5448feb5518c4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:32.784168 4885 consensus_queue.cc:1035] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:32.784310 4887 consensus_queue.cc:1035] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:32.814497 4283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:32.814644 4283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:32.816532 4283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: Sending DeleteTablet for 3 replicas of tablet 62756b8d0efd4fa39bb1a84e8958c457
I20260928 02:41:32.817153 4560 tablet_service.cc:1514] Processing DeleteTablet for tablet 62756b8d0efd4fa39bb1a84e8958c457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:32 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:44354
I20260928 02:41:32.817150 4700 tablet_service.cc:1514] Processing DeleteTablet for tablet 62756b8d0efd4fa39bb1a84e8958c457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:32 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:58498
I20260928 02:41:32.817433 4870 tablet_replica.cc:330] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: stopping tablet replica
I20260928 02:41:32.817571 4870 raft_consensus.cc:2229] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:32.817747 4870 raft_consensus.cc:2258] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:32.817705 4283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:41:32.817909 4868 tablet_replica.cc:330] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: stopping tablet replica
I20260928 02:41:32.818035 4868 raft_consensus.cc:2229] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 02:41:32.818102 4283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:32.818132 4868 raft_consensus.cc:2258] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:32.818722 4870 ts_tablet_manager.cc:1905] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:32.818828 4868 ts_tablet_manager.cc:1905] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:32.820838 4870 ts_tablet_manager.cc:1918] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:32.820902 4870 log.cc:1196] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/62756b8d0efd4fa39bb1a84e8958c457
I20260928 02:41:32.821120 4870 ts_tablet_manager.cc:1939] T 62756b8d0efd4fa39bb1a84e8958c457 P f3dee3aca5a04121a5448feb5518c4e3: Deleting consensus metadata
I20260928 02:41:32.821260 4868 ts_tablet_manager.cc:1918] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:32.821319 4868 log.cc:1196] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/62756b8d0efd4fa39bb1a84e8958c457
I20260928 02:41:32.821525 4868 ts_tablet_manager.cc:1939] T 62756b8d0efd4fa39bb1a84e8958c457 P 6cf1afa854654729a173460396cf9a04: Deleting consensus metadata
I20260928 02:41:32.821720 4270 catalog_manager.cc:4841] TS f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631): tablet 62756b8d0efd4fa39bb1a84e8958c457 (table test-table [id=7b8124063d3446c4bc31e93d0991fa9e]) successfully deleted
I20260928 02:41:32.822129 4270 catalog_manager.cc:4841] TS 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609): tablet 62756b8d0efd4fa39bb1a84e8958c457 (table test-table [id=7b8124063d3446c4bc31e93d0991fa9e]) successfully deleted
I20260928 02:41:32.822201 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet 62756b8d0efd4fa39bb1a84e8958c457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:32 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:55592
I20260928 02:41:32.822384 4869 tablet_replica.cc:330] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: stopping tablet replica
I20260928 02:41:32.822489 4869 raft_consensus.cc:2229] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:32.822599 4869 raft_consensus.cc:2258] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:32.823206 4869 ts_tablet_manager.cc:1905] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:32.825479 4869 ts_tablet_manager.cc:1918] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:32.825538 4869 log.cc:1196] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/62756b8d0efd4fa39bb1a84e8958c457
I20260928 02:41:32.825639 4700 tablet_service.cc:1467] Processing CreateTablet for tablet a49252b89a7f41a7a60603eddde0ac96 (DEFAULT_TABLE table=test-table [id=7c8427f0cc134684b25bf13dc153d2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:32.825749 4869 ts_tablet_manager.cc:1939] T 62756b8d0efd4fa39bb1a84e8958c457 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Deleting consensus metadata
I20260928 02:41:32.825739 4560 tablet_service.cc:1467] Processing CreateTablet for tablet a49252b89a7f41a7a60603eddde0ac96 (DEFAULT_TABLE table=test-table [id=7c8427f0cc134684b25bf13dc153d2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:32.825892 4700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49252b89a7f41a7a60603eddde0ac96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:32.825935 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49252b89a7f41a7a60603eddde0ac96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:32.826453 4420 tablet_service.cc:1467] Processing CreateTablet for tablet a49252b89a7f41a7a60603eddde0ac96 (DEFAULT_TABLE table=test-table [id=7c8427f0cc134684b25bf13dc153d2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:32.826653 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49252b89a7f41a7a60603eddde0ac96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:32.828027 4879 tablet_bootstrap.cc:492] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: Bootstrap starting.
I20260928 02:41:32.828042 4880 tablet_bootstrap.cc:492] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: Bootstrap starting.
I20260928 02:41:32.828737 4879 tablet_bootstrap.cc:654] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:32.828742 4881 tablet_bootstrap.cc:492] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Bootstrap starting.
I20260928 02:41:32.828821 4880 tablet_bootstrap.cc:654] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:32.829152 4270 catalog_manager.cc:4841] TS 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055): tablet 62756b8d0efd4fa39bb1a84e8958c457 (table test-table [id=7b8124063d3446c4bc31e93d0991fa9e]) successfully deleted
I20260928 02:41:32.829563 4881 tablet_bootstrap.cc:654] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:32.830017 4880 tablet_bootstrap.cc:492] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: No bootstrap required, opened a new log
I20260928 02:41:32.830098 4880 ts_tablet_manager.cc:1397] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:32.830386 4879 tablet_bootstrap.cc:492] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: No bootstrap required, opened a new log
I20260928 02:41:32.830456 4879 ts_tablet_manager.cc:1397] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:32.830476 4880 raft_consensus.cc:348] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.830572 4880 raft_consensus.cc:374] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:32.830601 4880 raft_consensus.cc:729] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3dee3aca5a04121a5448feb5518c4e3, State: Initialized, Role: FOLLOWER
I20260928 02:41:32.830681 4881 tablet_bootstrap.cc:492] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f: No bootstrap required, opened a new log
I20260928 02:41:32.830763 4881 ts_tablet_manager.cc:1397] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:32.830720 4880 consensus_queue.cc:260] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.830828 4879 raft_consensus.cc:348] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.830919 4879 raft_consensus.cc:374] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:32.830946 4879 raft_consensus.cc:729] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf1afa854654729a173460396cf9a04, State: Initialized, Role: FOLLOWER
I20260928 02:41:32.830988 4880 ts_tablet_manager.cc:1428] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:32.831063 4879 consensus_queue.cc:260] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.831132 4881 raft_consensus.cc:348] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.831225 4881 raft_consensus.cc:374] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:32.831259 4881 raft_consensus.cc:729] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4129dc7fcb4a07a5e1d7ac055c293f, State: Initialized, Role: FOLLOWER
I20260928 02:41:32.831306 4879 ts_tablet_manager.cc:1428] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:32.831363 4881 consensus_queue.cc:260] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:32.831596 4881 ts_tablet_manager.cc:1428] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:33.037808 4885 raft_consensus.cc:482] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:33.038012 4885 raft_consensus.cc:504] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:33.038518 4885 leader_election.cc:288] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055), 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609)
I20260928 02:41:33.039054 4439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49252b89a7f41a7a60603eddde0ac96" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" is_pre_election: true
I20260928 02:41:33.039111 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49252b89a7f41a7a60603eddde0ac96" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1afa854654729a173460396cf9a04" is_pre_election: true
I20260928 02:41:33.039213 4439 raft_consensus.cc:2454] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 0.
I20260928 02:41:33.039261 4580 raft_consensus.cc:2454] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 0.
I20260928 02:41:33.039510 4654 leader_election.cc:302] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4129dc7fcb4a07a5e1d7ac055c293f, f3dee3aca5a04121a5448feb5518c4e3; no voters:
I20260928 02:41:33.039672 4885 raft_consensus.cc:2781] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:33.039742 4885 raft_consensus.cc:482] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:33.039786 4885 raft_consensus.cc:3037] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:33.040791 4885 raft_consensus.cc:504] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:33.041121 4885 leader_election.cc:288] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055), 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609)
I20260928 02:41:33.041494 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49252b89a7f41a7a60603eddde0ac96" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1afa854654729a173460396cf9a04"
I20260928 02:41:33.041519 4439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49252b89a7f41a7a60603eddde0ac96" candidate_uuid: "f3dee3aca5a04121a5448feb5518c4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f"
I20260928 02:41:33.041607 4580 raft_consensus.cc:3037] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:33.041631 4439 raft_consensus.cc:3037] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:33.042570 4580 raft_consensus.cc:2454] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 1.
I20260928 02:41:33.042793 4439 raft_consensus.cc:2454] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3dee3aca5a04121a5448feb5518c4e3 in term 1.
I20260928 02:41:33.042846 4654 leader_election.cc:302] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf1afa854654729a173460396cf9a04, f3dee3aca5a04121a5448feb5518c4e3; no voters:
I20260928 02:41:33.042995 4885 raft_consensus.cc:2781] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:33.043074 4885 raft_consensus.cc:686] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 LEADER]: Becoming Leader. State: Replica: f3dee3aca5a04121a5448feb5518c4e3, State: Running, Role: LEADER
I20260928 02:41:33.043195 4885 consensus_queue.cc:237] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } }
I20260928 02:41:33.045006 4283 catalog_manager.cc:5495] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3). New cstate: current_term: 1 leader_uuid: "f3dee3aca5a04121a5448feb5518c4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3dee3aca5a04121a5448feb5518c4e3" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 39631 } health_report { overall_health: HEALTHY } } }
I20260928 02:41:33.082314 4580 raft_consensus.cc:1261] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Refusing update from remote peer f3dee3aca5a04121a5448feb5518c4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:33.082414 4439 raft_consensus.cc:1261] T a49252b89a7f41a7a60603eddde0ac96 P 3c4129dc7fcb4a07a5e1d7ac055c293f [term 1 FOLLOWER]: Refusing update from remote peer f3dee3aca5a04121a5448feb5518c4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:33.082685 4885 consensus_queue.cc:1035] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf1afa854654729a173460396cf9a04" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:33.082859 4895 consensus_queue.cc:1035] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4129dc7fcb4a07a5e1d7ac055c293f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:41:33.096714 4852 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:41:34.650116 4281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:34.650301 4281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:34.653482 4281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e93de74285734d19a771ca3065a6fc7f: Sending DeleteTablet for 3 replicas of tablet a49252b89a7f41a7a60603eddde0ac96
I20260928 02:41:34.654119 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet a49252b89a7f41a7a60603eddde0ac96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:34 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:55592
I20260928 02:41:34.654124 4700 tablet_service.cc:1514] Processing DeleteTablet for tablet a49252b89a7f41a7a60603eddde0ac96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:34 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:58498
I20260928 02:41:34.654129 4560 tablet_service.cc:1514] Processing DeleteTablet for tablet a49252b89a7f41a7a60603eddde0ac96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:34 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:44354
I20260928 02:41:34.654520 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 4355
I20260928 02:41:34.655269 4926 tablet_replica.cc:330] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: stopping tablet replica
I20260928 02:41:34.655467 4926 raft_consensus.cc:2229] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:34.655702 4926 raft_consensus.cc:2258] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:34.656433 4926 ts_tablet_manager.cc:1905] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:34.659121 4926 ts_tablet_manager.cc:1918] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:41:34.659224 4926 log.cc:1196] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/a49252b89a7f41a7a60603eddde0ac96
I20260928 02:41:34.659456 4926 ts_tablet_manager.cc:1939] T a49252b89a7f41a7a60603eddde0ac96 P 6cf1afa854654729a173460396cf9a04: Deleting consensus metadata
I20260928 02:41:34.660317 4270 catalog_manager.cc:4841] TS 6cf1afa854654729a173460396cf9a04 (127.2.123.2:33609): tablet a49252b89a7f41a7a60603eddde0ac96 (table test-table [id=7c8427f0cc134684b25bf13dc153d2eb]) successfully deleted
W20260928 02:41:34.660892 4270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:46055 (error 108)
I20260928 02:41:34.661392 4924 tablet_replica.cc:330] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: stopping tablet replica
I20260928 02:41:34.661528 4924 raft_consensus.cc:2229] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:34.661728 4924 raft_consensus.cc:2258] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:41:34.662266 4928 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:34.661591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:46055 (local address 127.0.0.1:55604)
0928 02:41:34.661822 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:34.661839 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:34.661954 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:34.662145 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:46055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20260928 02:41:34.662380 4924 ts_tablet_manager.cc:1905] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:34.662385 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 4495
W20260928 02:41:34.662371 4270 catalog_manager.cc:4584] TS 3c4129dc7fcb4a07a5e1d7ac055c293f (127.2.123.1:46055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a49252b89a7f41a7a60603eddde0ac96: Network error: Client connection negotiation failed: client connection to 127.2.123.1:46055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:41:34.665318 4924 ts_tablet_manager.cc:1918] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:41:34.665380 4924 log.cc:1196] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/a49252b89a7f41a7a60603eddde0ac96
I20260928 02:41:34.665620 4924 ts_tablet_manager.cc:1939] T a49252b89a7f41a7a60603eddde0ac96 P f3dee3aca5a04121a5448feb5518c4e3: Deleting consensus metadata
I20260928 02:41:34.666240 4270 catalog_manager.cc:4841] TS f3dee3aca5a04121a5448feb5518c4e3 (127.2.123.3:39631): tablet a49252b89a7f41a7a60603eddde0ac96 (table test-table [id=7c8427f0cc134684b25bf13dc153d2eb]) successfully deleted
I20260928 02:41:34.668133 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 4635
I20260928 02:41:34.673259 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 4251
2026-09-28T02:41:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8614 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:34 dist-test-slave-rqvb krb5kdc[4931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:34 dist-test-slave-rqvb krb5kdc[4931](info): set up 2 sockets
Sep 28 02:41:34 dist-test-slave-rqvb krb5kdc[4931](info): commencing operation
krb5kdc: starting...
W20260928 02:41:35.638939 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:35 dist-test-slave-rqvb krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:35.759248 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:46735
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:46735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:35.868175 4944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:35.868467 4944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:35.868533 4944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:35.873286 4944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:35.873371 4944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:35.873400 4944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:35.873425 4944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:35.873453 4944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:35.880589 4944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:46735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:35.881781 4944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:35.882896 4944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:35.892513 4949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:35.892493 4952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:35.892613 4944 server_base.cc:1034] running on GCE node
W20260928 02:41:35.892493 4950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:35.893577 4944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:35.894886 4944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:35.896080 4944 hybrid_clock.cc:648] HybridClock initialized: now 1790563295896062 us; error 34 us; skew 500 ppm
Sep 28 02:41:35 dist-test-slave-rqvb krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:35.898967 4944 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:35.899871 4944 webserver.cc:460] Webserver started at http://127.2.123.62:41775/ using document root <none> and password file <none>
I20260928 02:41:35.900384 4944 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:35.900442 4944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:35.900619 4944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:35.902760 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "46bc45bce18248a290499ca67c324af0"
format_stamp: "Formatted at 2026-09-28 02:41:35 on dist-test-slave-rqvb"
server_key: "63d6a2aa621621183a85ba1c36f77c52"
server_key_iv: "a2343c9303379640a2bbb6784d829f0e"
server_key_version: "encryptionkey@0"
I20260928 02:41:35.903283 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "46bc45bce18248a290499ca67c324af0"
format_stamp: "Formatted at 2026-09-28 02:41:35 on dist-test-slave-rqvb"
server_key: "63d6a2aa621621183a85ba1c36f77c52"
server_key_iv: "a2343c9303379640a2bbb6784d829f0e"
server_key_version: "encryptionkey@0"
I20260928 02:41:35.907076 4944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:41:35.909519 4959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:35.910844 4944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:35.910951 4944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "46bc45bce18248a290499ca67c324af0"
format_stamp: "Formatted at 2026-09-28 02:41:35 on dist-test-slave-rqvb"
server_key: "63d6a2aa621621183a85ba1c36f77c52"
server_key_iv: "a2343c9303379640a2bbb6784d829f0e"
server_key_version: "encryptionkey@0"
I20260928 02:41:35.911070 4944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:35.933617 4944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:35.936718 4944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:35.936873 4944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:35.943892 4944 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:46735
I20260928 02:41:35.943907 5011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:46735 every 8 connection(s)
I20260928 02:41:35.944830 4944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:35.945559 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 4944
I20260928 02:41:35.945670 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:35.945976 2540 external_mini_cluster.cc:1428] Setting key 49fc8880483c0b3210af90361cdd5678
I20260928 02:41:35.948259 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:35.954069 5012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0: Bootstrap starting.
Sep 28 02:41:35 dist-test-slave-rqvb krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:35.956516 5012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:35.957221 5012 log.cc:824] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:35.959455 5012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0: No bootstrap required, opened a new log
I20260928 02:41:35.960347 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:35.947534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58978 (local address 127.2.123.62:46735)
0928 02:41:35.948007 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:35.948023 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:35.948057 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:41:35.948790 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:35.948808 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:35.949149 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:35.949452 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:35.950257 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:35.951239 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:35.952089 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:35.952549 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:35.952687 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:35.955472 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:35.955477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:35.955490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:35.955583 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:35.957201 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:35.957645 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:35.957650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:35.957654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:35.957765 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:35.958108 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:35.958111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:35.958113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:35.958603 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:35.958800 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:35.959107 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:41:35.959410 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":161,"threads_started":1}
I20260928 02:41:35.963733 5012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46bc45bce18248a290499ca67c324af0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46735 } }
I20260928 02:41:35.963953 5012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:35.963985 5012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46bc45bce18248a290499ca67c324af0, State: Initialized, Role: FOLLOWER
I20260928 02:41:35.964380 5012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46bc45bce18248a290499ca67c324af0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46735 } }
I20260928 02:41:35.964498 5012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:35.964543 5012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:35.964653 5012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:35.965502 5012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46bc45bce18248a290499ca67c324af0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46735 } }
I20260928 02:41:35.965790 5012 leader_election.cc:302] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46bc45bce18248a290499ca67c324af0; no voters:
I20260928 02:41:35.966078 5012 leader_election.cc:288] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:35.966197 5017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:35.966393 5017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [term 1 LEADER]: Becoming Leader. State: Replica: 46bc45bce18248a290499ca67c324af0, State: Running, Role: LEADER
I20260928 02:41:35.966704 5017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46bc45bce18248a290499ca67c324af0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46735 } }
I20260928 02:41:35.967234 5012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:35.968281 5018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46bc45bce18248a290499ca67c324af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46bc45bce18248a290499ca67c324af0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46735 } } }
I20260928 02:41:35.968397 5018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:35.968685 5019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46bc45bce18248a290499ca67c324af0. Latest consensus state: current_term: 1 leader_uuid: "46bc45bce18248a290499ca67c324af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46bc45bce18248a290499ca67c324af0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46735 } } }
I20260928 02:41:35.968797 5019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:35.968757 5026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:35.971873 5026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:35.977741 5026 catalog_manager.cc:1347] Generated new cluster ID: 2246f61c17ff43e992043da952db461f
I20260928 02:41:35.977804 5026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:35.995420 5026 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:35.996563 5026 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:36.001340 5026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46bc45bce18248a290499ca67c324af0: Generated new TSK 0
I20260928 02:41:36.002146 5026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:36.034228 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46735
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:36.143911 5038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:36.144186 5038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:36.144229 5038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:36.148412 5038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:36.148483 5038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:36.148588 5038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:36.154620 5038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:36.155823 5038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:36.156916 5038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:36.166680 5044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:36.166702 5043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:36.166787 5038 server_base.cc:1034] running on GCE node
W20260928 02:41:36.166671 5046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:36.167462 5038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:36.168125 5038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:36.169296 5038 hybrid_clock.cc:648] HybridClock initialized: now 1790563296169270 us; error 44 us; skew 500 ppm
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:36.172170 5038 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:36.173043 5038 webserver.cc:460] Webserver started at http://127.2.123.1:37573/ using document root <none> and password file <none>
I20260928 02:41:36.173584 5038 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:36.173652 5038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:36.173885 5038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:36.176007 5038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "76f199df5a10463ea392238eca80a332"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "7b1ecc52e74ce1c3d562a40b6d39cab4"
server_key_iv: "63a8b66a658546299ed2512e0e1b87c5"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.176554 5038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "76f199df5a10463ea392238eca80a332"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "7b1ecc52e74ce1c3d562a40b6d39cab4"
server_key_iv: "63a8b66a658546299ed2512e0e1b87c5"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.180326 5038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:41:36.182812 5053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.184015 5038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:36.184123 5038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "76f199df5a10463ea392238eca80a332"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "7b1ecc52e74ce1c3d562a40b6d39cab4"
server_key_iv: "63a8b66a658546299ed2512e0e1b87c5"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.184228 5038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:36.200922 5038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:36.204324 5038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:36.204521 5038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:36.205152 5038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:36.206152 5038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:36.206219 5038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.206270 5038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:36.206295 5038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.215447 5038 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:41251
I20260928 02:41:36.215561 5166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:41251 every 8 connection(s)
I20260928 02:41:36.216418 5038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:36.220887 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 5038
I20260928 02:41:36.221019 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:36.221361 2540 external_mini_cluster.cc:1428] Setting key 5134e678cd66cbe9ff488e214713e09e
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:41:36.231303 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.218385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:58605 (local address 127.2.123.62:46735)
0928 02:41:36.218649 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:36.218653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:36.219489 (+ 836us) server_negotiation.cc:407] Connection header received
0928 02:41:36.220993 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:36.220997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:36.221061 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:36.221149 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:36.222728 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.223340 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.224945 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.225188 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.225277 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:36.227859 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:36.227862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:36.227864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:36.227888 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:36.229255 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:36.229749 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:36.229752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:36.229753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:36.229828 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:36.230199 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:36.230202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:36.230204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:36.230383 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:36.230465 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:36.230993 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 02:41:36.231135 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:41:36.232120 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.218804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:46735 (local address 127.2.123.1:58605)
0928 02:41:36.219312 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:36.219360 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:36.220680 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:36.221396 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:36.221407 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:36.221899 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:36.222542 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:36.222557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.223487 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.223490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:36.224818 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:36.224829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.225348 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.225352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:36.225514 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:36.226144 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:36.226160 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:36.227687 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:36.229370 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:36.229376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:36.229393 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:36.229648 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:36.229914 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:36.229917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:36.229919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:36.230110 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:36.230484 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:36.230491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:36.230751 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 02:41:36.230914 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 02:41:36.231253 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:41:36.233456 5167 heartbeater.cc:347] Connected to a master server at 127.2.123.62:46735
I20260928 02:41:36.233700 5167 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:36.234321 5167 heartbeater.cc:511] Master 127.2.123.62:46735 requested a full tablet report, sending...
I20260928 02:41:36.235929 4976 ts_manager.cc:194] Registered new tserver with Master: 76f199df5a10463ea392238eca80a332 (127.2.123.1:41251)
I20260928 02:41:36.237044 4976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:58605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:36.250344 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46735
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:36.360735 5172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:36.361016 5172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:36.361058 5172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:36.365350 5172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:36.365418 5172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:36.365478 5172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:36.371008 5172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:36.372030 5172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:36.373085 5172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:36.382908 5177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:36.382977 5172 server_base.cc:1034] running on GCE node
W20260928 02:41:36.382961 5180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:36.382920 5178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:36.383550 5172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:36.384182 5172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:36.385347 5172 hybrid_clock.cc:648] HybridClock initialized: now 1790563296385316 us; error 43 us; skew 500 ppm
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:36.388181 5172 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:36.389129 5172 webserver.cc:460] Webserver started at http://127.2.123.2:37145/ using document root <none> and password file <none>
I20260928 02:41:36.389684 5172 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:36.389750 5172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:36.389936 5172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:36.391984 5172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "38222360f37541fa92a13878ee7d8295"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "f3f81da0d8fbf18d938d26c293f18641"
server_key_iv: "274e58af2082ed2519ec12e403cc240c"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.392529 5172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "38222360f37541fa92a13878ee7d8295"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "f3f81da0d8fbf18d938d26c293f18641"
server_key_iv: "274e58af2082ed2519ec12e403cc240c"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.396289 5172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:41:36.398705 5187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.399933 5172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:36.400043 5172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "38222360f37541fa92a13878ee7d8295"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "f3f81da0d8fbf18d938d26c293f18641"
server_key_iv: "274e58af2082ed2519ec12e403cc240c"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.400146 5172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:36.415482 5172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:36.418726 5172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:36.418907 5172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:36.419476 5172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:36.420316 5172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:36.420362 5172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.420416 5172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:36.420446 5172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.430739 5172 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:46427
I20260928 02:41:36.430856 5300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:46427 every 8 connection(s)
I20260928 02:41:36.431670 5172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:36.436811 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 5172
I20260928 02:41:36.436909 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:36.437176 2540 external_mini_cluster.cc:1428] Setting key d9d2378af2d1dba7b9a70ce8b9dbac6b
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:41:36.447317 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.433491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:44347 (local address 127.2.123.62:46735)
0928 02:41:36.433639 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:36.433644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:36.434562 (+ 918us) server_negotiation.cc:407] Connection header received
0928 02:41:36.435935 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:36.435938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:36.435983 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:36.436056 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:36.437397 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.438083 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.439561 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.439985 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.440090 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:36.443254 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:36.443258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:36.443261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:36.443291 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:36.444970 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:36.445489 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:36.445493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:36.445495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:36.445585 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:36.445915 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:36.445919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:36.445921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:36.446196 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:36.446303 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:36.447048 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 02:41:36.447189 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:41:36.448210 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.433913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:46735 (local address 127.2.123.2:44347)
0928 02:41:36.434401 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:36.434444 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:36.435742 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:36.436224 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:36.436231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:36.436656 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:36.437230 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:36.437248 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.438207 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.438211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:36.439415 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:36.439426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.440159 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.440164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:36.440371 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:36.441073 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:36.441095 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:36.443052 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:36.445092 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:36.445097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:36.445111 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:36.445381 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:36.445679 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:36.445681 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:36.445683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:36.445821 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:36.446325 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:36.446332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:36.446686 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 02:41:36.446966 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 02:41:36.447300 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:41:36.449568 5301 heartbeater.cc:347] Connected to a master server at 127.2.123.62:46735
I20260928 02:41:36.449877 5301 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:36.450414 5301 heartbeater.cc:511] Master 127.2.123.62:46735 requested a full tablet report, sending...
I20260928 02:41:36.451537 4976 ts_manager.cc:194] Registered new tserver with Master: 38222360f37541fa92a13878ee7d8295 (127.2.123.2:46427)
I20260928 02:41:36.452105 4976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:44347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:36.466030 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46735
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:36.574378 5306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:36.574643 5306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:36.574685 5306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:36.578655 5306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:36.578715 5306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:36.578789 5306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:36.584239 5306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:36.585256 5306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:36.586264 5306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:36.595551 5311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:36.595728 5312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:36.595557 5314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:36.595809 5306 server_base.cc:1034] running on GCE node
I20260928 02:41:36.596277 5306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:36.596940 5306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:36.598073 5306 hybrid_clock.cc:648] HybridClock initialized: now 1790563296598065 us; error 47 us; skew 500 ppm
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:36.600806 5306 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:36.601709 5306 webserver.cc:460] Webserver started at http://127.2.123.3:43541/ using document root <none> and password file <none>
I20260928 02:41:36.602344 5306 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:36.602413 5306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:36.602597 5306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:36.604691 5306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "5b15c09ed74c4d1aa6b6dcda9001bf6d"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "8b02c4ffd2b3ab6858cff35c608606f6"
server_key_iv: "b140f6dbd9bb4247cc6a34683b50797e"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.605240 5306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "5b15c09ed74c4d1aa6b6dcda9001bf6d"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "8b02c4ffd2b3ab6858cff35c608606f6"
server_key_iv: "b140f6dbd9bb4247cc6a34683b50797e"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.608964 5306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:41:36.611443 5321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.612854 5306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:36.612960 5306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "5b15c09ed74c4d1aa6b6dcda9001bf6d"
format_stamp: "Formatted at 2026-09-28 02:41:36 on dist-test-slave-rqvb"
server_key: "8b02c4ffd2b3ab6858cff35c608606f6"
server_key_iv: "b140f6dbd9bb4247cc6a34683b50797e"
server_key_version: "encryptionkey@0"
I20260928 02:41:36.613080 5306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:36.632414 5306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:36.635438 5306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:36.635620 5306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:36.636198 5306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:36.637008 5306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:36.637054 5306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.637106 5306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:36.637135 5306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:36.645730 5306 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:32953
I20260928 02:41:36.645771 5434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:32953 every 8 connection(s)
I20260928 02:41:36.646634 5306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:36.652463 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 5306
I20260928 02:41:36.652585 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:36.652855 2540 external_mini_cluster.cc:1428] Setting key a128eed5f899814272e5d9764aac2cdc
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:36.660568 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.648525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:59061 (local address 127.2.123.62:46735)
0928 02:41:36.648715 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:36.648719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:36.649512 (+ 793us) server_negotiation.cc:407] Connection header received
0928 02:41:36.650886 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:36.650889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:36.650945 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:36.651023 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:36.652226 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.652990 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.654697 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.654929 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.655035 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:36.657394 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:36.657397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:36.657399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:36.657424 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:36.658595 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:36.659070 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:36.659073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:36.659074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:36.659154 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:36.659471 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:36.659474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:36.659476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:36.659658 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:36.659801 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:36.660291 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 02:41:36.660411 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 02:41:36.661373 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:36.648888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:46735 (local address 127.2.123.3:59061)
0928 02:41:36.649343 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:36.649385 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:36.650650 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:36.651212 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:36.651219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:36.651568 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:36.652072 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:36.652086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.653145 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.653150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:36.654507 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:36.654517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:36.655089 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:36.655094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:36.655290 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:36.655794 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:36.655810 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:36.657226 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:36.658710 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:36.658715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:36.658731 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:36.658970 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:36.659244 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:36.659248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:36.659250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:36.659377 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:36.659820 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:36.659826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:36.660056 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 02:41:36.660229 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 02:41:36.660544 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":97,"threads_started":1}
I20260928 02:41:36.662488 5435 heartbeater.cc:347] Connected to a master server at 127.2.123.62:46735
I20260928 02:41:36.662714 5435 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:36.663195 5435 heartbeater.cc:511] Master 127.2.123.62:46735 requested a full tablet report, sending...
I20260928 02:41:36.664218 4976 ts_manager.cc:194] Registered new tserver with Master: 5b15c09ed74c4d1aa6b6dcda9001bf6d (127.2.123.3:32953)
I20260928 02:41:36.664815 4976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:59061
I20260928 02:41:36.666929 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:41:36.678982 5015 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:36.677561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.2.123.62:46735)
0928 02:41:36.677711 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:36.677716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:36.677856 (+ 140us) server_negotiation.cc:407] Connection header received
0928 02:41:36.678014 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:36.678017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:36.678063 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:36.678129 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:36.678845 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58986: BlockingRecv error: recv got EOF from 127.0.0.1:58986 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:41:36.679906 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 5038
I20260928 02:41:36.684652 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 5172
I20260928 02:41:36.688966 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 5306
I20260928 02:41:36.693181 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 4944
2026-09-28T02:41:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2018 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[5447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[5447](info): set up 2 sockets
Sep 28 02:41:36 dist-test-slave-rqvb krb5kdc[5447](info): commencing operation
krb5kdc: starting...
W20260928 02:41:37.649767 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:37 dist-test-slave-rqvb krb5kdc[5447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:37.769470 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45159
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:45159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:37.879734 5460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:37.880206 5460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:37.880302 5460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:37.885802 5460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:37.885875 5460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:37.885895 5460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:37.885917 5460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:37.885933 5460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:37.891498 5460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:45159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:37.892620 5460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:37.893664 5460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:37.902141 5465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:37.902284 5460 server_base.cc:1034] running on GCE node
W20260928 02:41:37.902191 5466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:37.902191 5468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:37.902974 5460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:37.904103 5460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:37.905278 5460 hybrid_clock.cc:648] HybridClock initialized: now 1790563297905250 us; error 41 us; skew 500 ppm
Sep 28 02:41:37 dist-test-slave-rqvb krb5kdc[5447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563297, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:37.912809 5460 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:37.913748 5460 webserver.cc:460] Webserver started at http://127.2.123.62:35311/ using document root <none> and password file <none>
I20260928 02:41:37.914352 5460 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:37.914417 5460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:37.914630 5460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:37.916791 5460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "669967042a0342f0a130e32b6dff3f35"
format_stamp: "Formatted at 2026-09-28 02:41:37 on dist-test-slave-rqvb"
server_key: "5c76b83e8ec0827254e233fe94418ce2"
server_key_iv: "01a85c7d7f4afc30b1b75578e734edca"
server_key_version: "encryptionkey@0"
I20260928 02:41:37.917335 5460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "669967042a0342f0a130e32b6dff3f35"
format_stamp: "Formatted at 2026-09-28 02:41:37 on dist-test-slave-rqvb"
server_key: "5c76b83e8ec0827254e233fe94418ce2"
server_key_iv: "01a85c7d7f4afc30b1b75578e734edca"
server_key_version: "encryptionkey@0"
I20260928 02:41:37.921044 5460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:41:37.923545 5475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:37.924862 5460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:37.924960 5460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "669967042a0342f0a130e32b6dff3f35"
format_stamp: "Formatted at 2026-09-28 02:41:37 on dist-test-slave-rqvb"
server_key: "5c76b83e8ec0827254e233fe94418ce2"
server_key_iv: "01a85c7d7f4afc30b1b75578e734edca"
server_key_version: "encryptionkey@0"
I20260928 02:41:37.925048 5460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:37.941015 5460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:37.943938 5460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:37.944092 5460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:37.950598 5460 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:45159
I20260928 02:41:37.950851 5527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:45159 every 8 connection(s)
I20260928 02:41:37.951534 5460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:37.954305 5528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:37.955404 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 5460
I20260928 02:41:37.955500 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:37.955761 2540 external_mini_cluster.cc:1428] Setting key 765c9214a4eaa8587ec819d4be6ba6c8
I20260928 02:41:37.960819 5528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35: Bootstrap starting.
I20260928 02:41:37.963439 5528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:37.964237 5528 log.cc:824] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:41:37 dist-test-slave-rqvb krb5kdc[5447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:37.966400 5528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35: No bootstrap required, opened a new log
I20260928 02:41:37.971057 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:37.957076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.2.123.62:45159)
0928 02:41:37.957516 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:37.957536 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:37.957579 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:41:37.958398 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:37.958421 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:37.958835 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:37.959214 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:37.959922 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:37.961319 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:37.962406 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:37.963089 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:37.963243 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:37.965768 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:37.965774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:37.965791 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:37.965822 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:37.967841 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:37.968294 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:37.968300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:37.968306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:37.968457 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:37.968758 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:37.968761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:37.968763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:37.969297 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:37.969484 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:37.969737 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:41:37.970085 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260928 02:41:37.970911 5528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669967042a0342f0a130e32b6dff3f35" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45159 } }
I20260928 02:41:37.971172 5528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:37.971216 5528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669967042a0342f0a130e32b6dff3f35, State: Initialized, Role: FOLLOWER
I20260928 02:41:37.971714 5528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669967042a0342f0a130e32b6dff3f35" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45159 } }
I20260928 02:41:37.971864 5528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:37.971915 5528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:37.972038 5528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:37.973018 5528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669967042a0342f0a130e32b6dff3f35" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45159 } }
I20260928 02:41:37.973373 5528 leader_election.cc:302] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 669967042a0342f0a130e32b6dff3f35; no voters:
I20260928 02:41:37.973733 5528 leader_election.cc:288] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:37.973860 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:41:37.973832 5533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:37.974082 5533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [term 1 LEADER]: Becoming Leader. State: Replica: 669967042a0342f0a130e32b6dff3f35, State: Running, Role: LEADER
I20260928 02:41:37.974439 5533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669967042a0342f0a130e32b6dff3f35" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45159 } }
I20260928 02:41:37.974972 5528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:37.975631 5533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 669967042a0342f0a130e32b6dff3f35. Latest consensus state: current_term: 1 leader_uuid: "669967042a0342f0a130e32b6dff3f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669967042a0342f0a130e32b6dff3f35" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45159 } } }
I20260928 02:41:37.976190 5534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "669967042a0342f0a130e32b6dff3f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669967042a0342f0a130e32b6dff3f35" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45159 } } }
I20260928 02:41:37.976284 5533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:37.976346 5534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:37.976712 5545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:37.979344 5545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:37.984587 5545 catalog_manager.cc:1347] Generated new cluster ID: 71618a616b1148daa090a0cfc7d65397
I20260928 02:41:37.984637 5545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:37.995939 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:37.990441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.2.123.62:45159)
0928 02:41:37.990549 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:37.990553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:37.990813 (+ 260us) server_negotiation.cc:407] Connection header received
0928 02:41:37.990936 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:37.990940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:37.991002 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:37.991090 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:37.991628 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:37.992425 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:37.993541 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:37.993916 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:37.994041 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:37.994904 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:37.994908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:37.994910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:37.995495 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:37.995530 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:37.995549 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:37.995635 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:37.995730 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 02:41:37.995789 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:41:38.006448 5545 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:38.007440 5545 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:38.017772 5545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 669967042a0342f0a130e32b6dff3f35: Generated new TSK 0
I20260928 02:41:38.018705 5545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:41:38.025182 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 5460
2026-09-28T02:41:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1331 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:38 dist-test-slave-rqvb krb5kdc[5561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:38 dist-test-slave-rqvb krb5kdc[5561](info): set up 2 sockets
Sep 28 02:41:38 dist-test-slave-rqvb krb5kdc[5561](info): commencing operation
krb5kdc: starting...
W20260928 02:41:38.981787 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:39 dist-test-slave-rqvb krb5kdc[5561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:39.102548 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:35687
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:35687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:39.212687 5574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:39.212990 5574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:39.213042 5574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:39.217165 5574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:39.217242 5574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:39.217260 5574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:39.217276 5574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:39.217291 5574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:39.222924 5574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:35687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:39.223989 5574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:39.225003 5574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:39.233366 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:39.233461 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:39.233497 5574 server_base.cc:1034] running on GCE node
W20260928 02:41:39.233367 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:39.234380 5574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:39.235459 5574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:39.236616 5574 hybrid_clock.cc:648] HybridClock initialized: now 1790563299236594 us; error 36 us; skew 500 ppm
Sep 28 02:41:39 dist-test-slave-rqvb krb5kdc[5561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:39.239392 5574 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:39.240270 5574 webserver.cc:460] Webserver started at http://127.2.123.62:34989/ using document root <none> and password file <none>
I20260928 02:41:39.240792 5574 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:39.240851 5574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:39.241101 5574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:39.243222 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "ca84c019b61e4e4da9e2252dac5599ab"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "fe6f5667620b37eb7b67ec13314e28cd"
server_key_iv: "b468b02c8aa312b789a0002e223b7708"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.243758 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "ca84c019b61e4e4da9e2252dac5599ab"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "fe6f5667620b37eb7b67ec13314e28cd"
server_key_iv: "b468b02c8aa312b789a0002e223b7708"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.247529 5574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:41:39.250185 5589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.251472 5574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:39.251578 5574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "ca84c019b61e4e4da9e2252dac5599ab"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "fe6f5667620b37eb7b67ec13314e28cd"
server_key_iv: "b468b02c8aa312b789a0002e223b7708"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.251691 5574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:39.270684 5574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:39.273826 5574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:39.274013 5574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:39.281237 5574 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:35687
I20260928 02:41:39.281504 5641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:35687 every 8 connection(s)
I20260928 02:41:39.282266 5574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:39.285132 5642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:39.289057 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 5574
I20260928 02:41:39.289175 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:39.289467 2540 external_mini_cluster.cc:1428] Setting key d4457c4d48211dc1514dc6391b6402e7
I20260928 02:41:39.291539 5642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab: Bootstrap starting.
I20260928 02:41:39.293922 5642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:39.294694 5642 log.cc:824] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:39.296527 5642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab: No bootstrap required, opened a new log
Sep 28 02:41:39 dist-test-slave-rqvb krb5kdc[5561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:39.300302 5642 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca84c019b61e4e4da9e2252dac5599ab" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35687 } }
I20260928 02:41:39.300549 5642 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:39.300585 5642 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca84c019b61e4e4da9e2252dac5599ab, State: Initialized, Role: FOLLOWER
I20260928 02:41:39.301186 5642 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca84c019b61e4e4da9e2252dac5599ab" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35687 } }
I20260928 02:41:39.301393 5642 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:39.301479 5642 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:39.301589 5642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:39.302734 5642 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca84c019b61e4e4da9e2252dac5599ab" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35687 } }
I20260928 02:41:39.303090 5642 leader_election.cc:302] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca84c019b61e4e4da9e2252dac5599ab; no voters:
I20260928 02:41:39.303395 5642 leader_election.cc:288] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:39.303488 5647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:39.303654 5647 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [term 1 LEADER]: Becoming Leader. State: Replica: ca84c019b61e4e4da9e2252dac5599ab, State: Running, Role: LEADER
I20260928 02:41:39.303962 5647 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca84c019b61e4e4da9e2252dac5599ab" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35687 } }
I20260928 02:41:39.303997 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:39.291028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.2.123.62:35687)
0928 02:41:39.291483 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:39.291499 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:39.291535 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:41:39.292225 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:39.292243 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:39.292583 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:39.292906 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:39.293705 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.294797 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.295640 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.296125 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.296270 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:39.298897 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:39.298902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:39.298916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:39.299015 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:39.300673 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:39.301170 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:39.301175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:39.301179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:39.301313 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:39.301602 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:39.301604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:39.301606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:39.302128 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:39.302285 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:39.302655 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:41:39.302972 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260928 02:41:39.304719 5642 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:39.305661 5649 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca84c019b61e4e4da9e2252dac5599ab. Latest consensus state: current_term: 1 leader_uuid: "ca84c019b61e4e4da9e2252dac5599ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca84c019b61e4e4da9e2252dac5599ab" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35687 } } }
I20260928 02:41:39.305792 5649 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:39.306156 5648 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca84c019b61e4e4da9e2252dac5599ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca84c019b61e4e4da9e2252dac5599ab" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35687 } } }
I20260928 02:41:39.306245 5648 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab [sys.catalog]: This master's current role is: LEADER
W20260928 02:41:39.309302 5662 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:41:39.309373 5662 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:41:39.309471 5656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:39.310490 5656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:39.316387 5656 catalog_manager.cc:1347] Generated new cluster ID: 0bf0818253854a7bb01a8c8db657ab5c
I20260928 02:41:39.316464 5656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:39.327260 5656 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:39.328255 5656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:39.333212 5656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca84c019b61e4e4da9e2252dac5599ab: Generated new TSK 0
I20260928 02:41:39.334067 5656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:39.374430 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35687
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:39.512060 5668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:39.512327 5668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:39.512370 5668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:39.516372 5668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:39.516438 5668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:39.516496 5668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:39.521798 5668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:39.522945 5668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:39.523962 5668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:39.533195 5673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:39.533238 5676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:39.533252 5674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:39.533362 5668 server_base.cc:1034] running on GCE node
I20260928 02:41:39.533756 5668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:39.534457 5668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:39.535626 5668 hybrid_clock.cc:648] HybridClock initialized: now 1790563299535606 us; error 28 us; skew 500 ppm
Sep 28 02:41:39 dist-test-slave-rqvb krb5kdc[5561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:39.538529 5668 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:39.539390 5668 webserver.cc:460] Webserver started at http://127.2.123.1:44711/ using document root <none> and password file <none>
I20260928 02:41:39.539907 5668 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:39.539973 5668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:39.540144 5668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:39.542307 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "861238cb3a6040c29d156d645ab58c83"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "c3001073322ee3c0397a6d72512e62bd"
server_key_iv: "6e9d20566922324c0d81c6da3fc2e589"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.542840 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "861238cb3a6040c29d156d645ab58c83"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "c3001073322ee3c0397a6d72512e62bd"
server_key_iv: "6e9d20566922324c0d81c6da3fc2e589"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.546692 5668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:41:39.549364 5683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.550808 5668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:39.550916 5668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "861238cb3a6040c29d156d645ab58c83"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "c3001073322ee3c0397a6d72512e62bd"
server_key_iv: "6e9d20566922324c0d81c6da3fc2e589"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.551040 5668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:39.573315 5668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:39.576344 5668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:39.576521 5668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:39.577091 5668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:39.577863 5668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:39.577908 5668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.577945 5668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:39.578009 5668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.586309 5668 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:42235
I20260928 02:41:39.586329 5796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:42235 every 8 connection(s)
I20260928 02:41:39.587218 5668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:39.591902 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 5668
I20260928 02:41:39.592008 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:39.592284 2540 external_mini_cluster.cc:1428] Setting key e92a3a591804c9ea135047587b044897
Sep 28 02:41:39 dist-test-slave-rqvb krb5kdc[5561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:39.602093 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:39.589123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:53099 (local address 127.2.123.62:35687)
0928 02:41:39.589250 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:39.589253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:39.590168 (+ 915us) server_negotiation.cc:407] Connection header received
0928 02:41:39.591485 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:39.591489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:39.591545 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:39.591618 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:39.592898 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.593660 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.595300 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.595567 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.595663 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:39.598074 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:39.598078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:39.598081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:39.598117 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:39.599800 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:39.600331 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:39.600336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:39.600338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:39.600447 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:39.600847 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:39.600851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:39.600852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:39.601090 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:39.601201 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:39.601730 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 02:41:39.601869 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:41:39.602799 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:39.589510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35687 (local address 127.2.123.1:53099)
0928 02:41:39.589943 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:39.590025 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:39.591274 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:39.591828 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:39.591836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:39.592234 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:39.592742 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:39.592762 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.593790 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.593794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:39.595166 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:39.595177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.595716 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.595720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:39.595892 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:39.596506 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:39.596522 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:39.597902 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:39.599923 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:39.599927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:39.599940 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:39.600194 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:39.600572 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:39.600575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:39.600577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:39.600729 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:39.601193 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:39.601198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:39.601445 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 02:41:39.601596 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:41:39.601869 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260928 02:41:39.604111 5797 heartbeater.cc:347] Connected to a master server at 127.2.123.62:35687
I20260928 02:41:39.604362 5797 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:39.604884 5797 heartbeater.cc:511] Master 127.2.123.62:35687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:41:39.606447 5606 ts_manager.cc:194] Registered new tserver with Master: 861238cb3a6040c29d156d645ab58c83 (127.2.123.1:42235)
I20260928 02:41:39.607889 5606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:53099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:39.621759 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35687
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:39.730863 5802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:39.731101 5802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:39.731158 5802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:39.735057 5802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:39.735110 5802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:39.735188 5802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:39.740269 5802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:39.741225 5802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:39.742233 5802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:39.751528 5810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:39.751570 5808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:39.751538 5807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:39.751713 5802 server_base.cc:1034] running on GCE node
I20260928 02:41:39.752301 5802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:39.752956 5802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:39.754148 5802 hybrid_clock.cc:648] HybridClock initialized: now 1790563299754120 us; error 41 us; skew 500 ppm
Sep 28 02:41:39 dist-test-slave-rqvb krb5kdc[5561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:39.757112 5802 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:39.758100 5802 webserver.cc:460] Webserver started at http://127.2.123.2:37205/ using document root <none> and password file <none>
I20260928 02:41:39.758662 5802 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:39.758737 5802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:39.758944 5802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:39.761018 5802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "259935ca53f742999c193bbf3e87c1a5"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "267ee139456f878ac74d81af6754fcb9"
server_key_iv: "e44387fb61a95c7244d6155b6458cdd3"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.761579 5802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "259935ca53f742999c193bbf3e87c1a5"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "267ee139456f878ac74d81af6754fcb9"
server_key_iv: "e44387fb61a95c7244d6155b6458cdd3"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.765424 5802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 02:41:39.767920 5817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.769219 5802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:39.769328 5802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "259935ca53f742999c193bbf3e87c1a5"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "267ee139456f878ac74d81af6754fcb9"
server_key_iv: "e44387fb61a95c7244d6155b6458cdd3"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.769447 5802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:39.782557 5802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:39.785427 5802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:39.785604 5802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:39.786197 5802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:39.786981 5802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:39.787026 5802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.787078 5802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:39.787107 5802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.795411 5802 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:42141
I20260928 02:41:39.795446 5930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:42141 every 8 connection(s)
I20260928 02:41:39.796319 5802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:39.797672 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 5802
I20260928 02:41:39.797775 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:39.798120 2540 external_mini_cluster.cc:1428] Setting key 0c54cb136f45ada0ed67ab854d7ed693
Sep 28 02:41:39 dist-test-slave-rqvb krb5kdc[5561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:41:39.810911 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:39.798145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:55931 (local address 127.2.123.62:35687)
0928 02:41:39.798285 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:39.798289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:39.799185 (+ 896us) server_negotiation.cc:407] Connection header received
0928 02:41:39.800686 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:39.800689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:39.800739 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:39.800821 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:39.802071 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.802859 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.804211 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.804435 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.804520 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:39.807520 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:39.807524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:39.807527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:39.807551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:39.808858 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:39.809370 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:39.809373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:39.809374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:39.809456 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:39.809785 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:39.809788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:39.809790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:39.810001 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:39.810091 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:39.810625 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 02:41:39.810749 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:41:39.811827 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:39.798497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35687 (local address 127.2.123.2:55931)
0928 02:41:39.799011 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:39.799068 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:39.800454 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:39.800974 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:39.800981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:39.801347 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:39.801869 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:39.801890 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.803021 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.803025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:39.804079 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:39.804090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:39.804612 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:39.804617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:39.804831 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:39.805535 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:39.805556 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:39.807341 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:39.808984 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:39.808990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:39.809008 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:39.809257 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:39.809544 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:39.809546 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:39.809549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:39.809685 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:39.810097 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:39.810104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:39.810359 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 02:41:39.810555 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 02:41:39.810889 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20260928 02:41:39.813089 5931 heartbeater.cc:347] Connected to a master server at 127.2.123.62:35687
I20260928 02:41:39.813345 5931 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:39.813877 5931 heartbeater.cc:511] Master 127.2.123.62:35687 requested a full tablet report, sending...
I20260928 02:41:39.815069 5606 ts_manager.cc:194] Registered new tserver with Master: 259935ca53f742999c193bbf3e87c1a5 (127.2.123.2:42141)
I20260928 02:41:39.815680 5606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:55931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:39.827363 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35687
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:39.939329 5936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:39.939584 5936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:39.939623 5936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:39.943583 5936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:39.943638 5936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:39.943693 5936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:39.948830 5936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:39.949769 5936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:39.950793 5936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:39.960551 5941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:39.960579 5936 server_base.cc:1034] running on GCE node
W20260928 02:41:39.960531 5944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:39.960496 5942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:39.961184 5936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:39.961817 5936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:39.963023 5936 hybrid_clock.cc:648] HybridClock initialized: now 1790563299962945 us; error 42 us; skew 500 ppm
Sep 28 02:41:39 dist-test-slave-rqvb krb5kdc[5561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:39.965690 5936 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:39.966675 5936 webserver.cc:460] Webserver started at http://127.2.123.3:44225/ using document root <none> and password file <none>
I20260928 02:41:39.967207 5936 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:39.967265 5936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:39.967458 5936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:39.969463 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "cbadd0e89de740bdba9f0fa9e735c7f5"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "e17aeb2f7b18a09d9474a57199e3435b"
server_key_iv: "0e098ac99ad21a3c30453a4a3afa5a35"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.970041 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "cbadd0e89de740bdba9f0fa9e735c7f5"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "e17aeb2f7b18a09d9474a57199e3435b"
server_key_iv: "0e098ac99ad21a3c30453a4a3afa5a35"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.973675 5936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:39.976085 5951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.977327 5936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:39.977437 5936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "cbadd0e89de740bdba9f0fa9e735c7f5"
format_stamp: "Formatted at 2026-09-28 02:41:39 on dist-test-slave-rqvb"
server_key: "e17aeb2f7b18a09d9474a57199e3435b"
server_key_iv: "0e098ac99ad21a3c30453a4a3afa5a35"
server_key_version: "encryptionkey@0"
I20260928 02:41:39.977545 5936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:39.990454 5936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:39.993391 5936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:39.993589 5936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:39.994297 5936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:39.995146 5936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:39.995196 5936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:39.995253 5936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:39.995286 5936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:40.004021 5936 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:46661
I20260928 02:41:40.004058 6064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:46661 every 8 connection(s)
I20260928 02:41:40.004948 5936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:40.013988 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 5936
I20260928 02:41:40.014117 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:40.014389 2540 external_mini_cluster.cc:1428] Setting key cb50c10551328ab7be5e8f5bb3c96971
Sep 28 02:41:40 dist-test-slave-rqvb krb5kdc[5561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563299, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:40.019469 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:40.006992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:32769 (local address 127.2.123.62:35687)
0928 02:41:40.007172 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:40.007175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:40.008087 (+ 912us) server_negotiation.cc:407] Connection header received
0928 02:41:40.009445 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:40.009449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:40.009506 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:40.009585 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:40.010843 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:40.011409 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:40.012590 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:40.012823 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:40.012913 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:40.015675 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:40.015678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:40.015680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:40.015709 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:40.017382 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:40.017911 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:40.017914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:40.017915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:40.018110 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:40.018453 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:40.018456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:40.018457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:40.018645 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:40.018717 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:40.019242 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 02:41:40.019347 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:41:40.020349 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:40.007384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35687 (local address 127.2.123.3:32769)
0928 02:41:40.007856 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:40.007901 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:40.009208 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:40.009784 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:40.009791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:40.010189 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:40.010684 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:40.010698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:40.011546 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:40.011549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:40.012449 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:40.012457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:40.012984 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:40.012988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:40.013162 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:40.013775 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:40.013792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:40.015471 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:40.017503 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:40.017508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:40.017522 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:40.017794 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:40.018212 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:40.018215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:40.018216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:40.018371 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:40.018725 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:40.018731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:40.018981 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 02:41:40.019160 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 02:41:40.019430 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260928 02:41:40.021585 6065 heartbeater.cc:347] Connected to a master server at 127.2.123.62:35687
I20260928 02:41:40.021850 6065 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:40.022387 6065 heartbeater.cc:511] Master 127.2.123.62:35687 requested a full tablet report, sending...
I20260928 02:41:40.023566 5606 ts_manager.cc:194] Registered new tserver with Master: cbadd0e89de740bdba9f0fa9e735c7f5 (127.2.123.3:46661)
I20260928 02:41:40.024189 5606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:32769
I20260928 02:41:40.029718 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:41:40 dist-test-slave-rqvb krb5kdc[5561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563300, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:41:40 dist-test-slave-rqvb krb5kdc[5561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563300, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:40.056015 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:40.047278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.2.123.62:35687)
0928 02:41:40.047427 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:40.047431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:40.047577 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:41:40.047685 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:40.047687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:40.047730 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:40.047798 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:40.048436 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:40.049018 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:40.049928 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:40.050315 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:40.050410 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:40.052835 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:40.052838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:40.052840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:40.052868 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:40.054194 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:40.054647 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:40.054650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:40.054652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:40.054762 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:40.055100 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:40.055103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:40.055105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:40.055358 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:40.055470 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:40.055741 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:41:40.055859 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:41:40.056602 5606 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35312
I20260928 02:41:40.057413 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 5668
I20260928 02:41:40.062156 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 5802
I20260928 02:41:40.066449 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 5936
I20260928 02:41:40.070585 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 5574
2026-09-28T02:41:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2045 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:40 dist-test-slave-rqvb krb5kdc[6077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:40 dist-test-slave-rqvb krb5kdc[6077](info): set up 2 sockets
Sep 28 02:41:40 dist-test-slave-rqvb krb5kdc[6077](info): commencing operation
krb5kdc: starting...
W20260928 02:41:41.021811 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:41 dist-test-slave-rqvb krb5kdc[6077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:41.148712 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:39921
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:39921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:41.257266 6090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:41.257546 6090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:41.257586 6090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:41.261616 6090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:41.261686 6090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:41.261703 6090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:41.261718 6090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:41.261732 6090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:41.267031 6090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:39921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:39921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:41.268067 6090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:41.269028 6090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:41.277236 6098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.277325 6096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.277276 6095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.277456 6090 server_base.cc:1034] running on GCE node
I20260928 02:41:41.278267 6090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:41.279479 6090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:41.280659 6090 hybrid_clock.cc:648] HybridClock initialized: now 1790563301280640 us; error 30 us; skew 500 ppm
Sep 28 02:41:41 dist-test-slave-rqvb krb5kdc[6077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:41.283432 6090 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:41.284299 6090 webserver.cc:460] Webserver started at http://127.2.123.62:37309/ using document root <none> and password file <none>
I20260928 02:41:41.284808 6090 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:41.284870 6090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:41.285040 6090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:41.287030 6090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "10243f4b0f14482cac8f7cba71d01f03"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "71cc14309833ecc75d5b7790a78e2acf"
server_key_iv: "ba6854ec7818b9e5d48ebbf1ef126332"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.287526 6090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "10243f4b0f14482cac8f7cba71d01f03"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "71cc14309833ecc75d5b7790a78e2acf"
server_key_iv: "ba6854ec7818b9e5d48ebbf1ef126332"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.291085 6090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:41:41.293401 6105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.294701 6090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:41.294803 6090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "10243f4b0f14482cac8f7cba71d01f03"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "71cc14309833ecc75d5b7790a78e2acf"
server_key_iv: "ba6854ec7818b9e5d48ebbf1ef126332"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.294962 6090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:41.315850 6090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:41.319294 6090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:41.319479 6090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:41.326670 6090 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:39921
I20260928 02:41:41.326690 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:39921 every 8 connection(s)
I20260928 02:41:41.327615 6090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:41.330564 6158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:41.335203 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 6090
I20260928 02:41:41.335338 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:41.335631 2540 external_mini_cluster.cc:1428] Setting key 5be63e1ab219c6ed77715dba8da400e5
I20260928 02:41:41.336434 6158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03: Bootstrap starting.
I20260928 02:41:41.338764 6158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:41.339480 6158 log.cc:824] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:41.341213 6158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03: No bootstrap required, opened a new log
Sep 28 02:41:41 dist-test-slave-rqvb krb5kdc[6077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:41.344576 6158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10243f4b0f14482cac8f7cba71d01f03" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39921 } }
I20260928 02:41:41.344802 6158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:41.344849 6158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10243f4b0f14482cac8f7cba71d01f03, State: Initialized, Role: FOLLOWER
I20260928 02:41:41.345296 6158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10243f4b0f14482cac8f7cba71d01f03" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39921 } }
I20260928 02:41:41.345455 6158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:41.345523 6158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:41.345618 6158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:41.346509 6158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10243f4b0f14482cac8f7cba71d01f03" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39921 } }
I20260928 02:41:41.346861 6158 leader_election.cc:302] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10243f4b0f14482cac8f7cba71d01f03; no voters:
I20260928 02:41:41.347213 6158 leader_election.cc:288] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:41.347298 6163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:41.347584 6163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [term 1 LEADER]: Becoming Leader. State: Replica: 10243f4b0f14482cac8f7cba71d01f03, State: Running, Role: LEADER
I20260928 02:41:41.347947 6163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10243f4b0f14482cac8f7cba71d01f03" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39921 } }
I20260928 02:41:41.348575 6158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:41.349210 6163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10243f4b0f14482cac8f7cba71d01f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10243f4b0f14482cac8f7cba71d01f03" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39921 } } }
I20260928 02:41:41.349404 6163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:41.349663 6163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10243f4b0f14482cac8f7cba71d01f03. Latest consensus state: current_term: 1 leader_uuid: "10243f4b0f14482cac8f7cba71d01f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10243f4b0f14482cac8f7cba71d01f03" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39921 } } }
I20260928 02:41:41.349794 6163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:41.350227 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.337094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.2.123.62:39921)
0928 02:41:41.337535 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.337552 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.337588 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:41:41.338285 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.338304 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.338653 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.338972 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:41.339950 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.340921 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.341810 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.342313 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.342457 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.344781 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:41.344786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:41.344799 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:41.344833 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:41.346615 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.347089 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.347095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.347101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.347238 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.347554 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.347557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.347559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.348055 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:41.348220 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.348918 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.349205 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":125,"threads_started":1}
W20260928 02:41:41.353404 6178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:41:41.353463 6178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:41:41.353543 6172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:41.354408 6172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:41.359494 6172 catalog_manager.cc:1347] Generated new cluster ID: 455e8f3d60314f3ba4328fdb5948fbe2
I20260928 02:41:41.359555 6172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:41.371640 6172 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:41.372596 6172 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:41.382915 6172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10243f4b0f14482cac8f7cba71d01f03: Generated new TSK 0
I20260928 02:41:41.383659 6172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:41.413271 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39921
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:41.547173 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:41.547448 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:41.547492 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:41.551719 6184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:41.551780 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:41.551841 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:41.557137 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:41.558225 6184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:41.559207 6184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:41.568598 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.568653 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.568753 6184 server_base.cc:1034] running on GCE node
W20260928 02:41:41.568779 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.569180 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:41.569819 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:41.570967 6184 hybrid_clock.cc:648] HybridClock initialized: now 1790563301570933 us; error 50 us; skew 500 ppm
Sep 28 02:41:41 dist-test-slave-rqvb krb5kdc[6077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:41.573563 6184 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:41.574394 6184 webserver.cc:460] Webserver started at http://127.2.123.1:34179/ using document root <none> and password file <none>
I20260928 02:41:41.574879 6184 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:41.574934 6184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:41.575085 6184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:41.576963 6184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "b88f729eade640fb8a6f703749382c3f"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "25a555fb1f6d67a52cbacde189c0bfcf"
server_key_iv: "8abd2bfc0418250042511cc6c4d4e899"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.577442 6184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "b88f729eade640fb8a6f703749382c3f"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "25a555fb1f6d67a52cbacde189c0bfcf"
server_key_iv: "8abd2bfc0418250042511cc6c4d4e899"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.580854 6184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:41:41.583277 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.584514 6184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:41.584619 6184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "b88f729eade640fb8a6f703749382c3f"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "25a555fb1f6d67a52cbacde189c0bfcf"
server_key_iv: "8abd2bfc0418250042511cc6c4d4e899"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.584717 6184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:41.600008 6184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:41.602896 6184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:41.603066 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:41.603613 6184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:41.604385 6184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:41.604429 6184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.604480 6184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:41.604507 6184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.612625 6184 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:39489
I20260928 02:41:41.612746 6312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:39489 every 8 connection(s)
I20260928 02:41:41.613485 6184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:41.620193 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 6184
I20260928 02:41:41.620294 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:41.620572 2540 external_mini_cluster.cc:1428] Setting key 0f8f7fd135474d8f0690e7cba3ea95e5
Sep 28 02:41:41 dist-test-slave-rqvb krb5kdc[6077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:41.626825 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.615305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:39561 (local address 127.2.123.62:39921)
0928 02:41:41.615457 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.615462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.616288 (+ 826us) server_negotiation.cc:407] Connection header received
0928 02:41:41.617518 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.617521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.617567 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.617646 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:41.618823 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.619338 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.620547 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.620899 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.621009 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.623364 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:41.623367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:41.623369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:41.623391 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:41.624721 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.625209 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.625213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.625215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.625309 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.625613 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.625616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.625618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.625840 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:41.625945 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.626535 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.626660 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:41:41.627599 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.615687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39921 (local address 127.2.123.1:39561)
0928 02:41:41.616110 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:41.616149 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:41.617315 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:41.617824 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:41.617831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:41.618216 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:41.618672 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.618685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.619453 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.619457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.620433 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.620441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.621061 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.621065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.621233 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.621786 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:41.621802 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:41.623205 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:41.624831 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.624837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.624852 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.625084 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.625400 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.625402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.625403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.625527 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.626000 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:41.626007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:41.626253 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 02:41:41.626460 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 02:41:41.626757 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260928 02:41:41.628767 6313 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39921
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:41:41.628979 6313 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:41.629428 6313 heartbeater.cc:511] Master 127.2.123.62:39921 requested a full tablet report, sending...
I20260928 02:41:41.631049 6122 ts_manager.cc:194] Registered new tserver with Master: b88f729eade640fb8a6f703749382c3f (127.2.123.1:39489)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:41:41.632437 6122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:39561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:41.649499 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39921
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:41.756392 6318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:41.756654 6318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:41.756711 6318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:41.760751 6318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:41.760814 6318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:41.760902 6318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:41.766371 6318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:41.767398 6318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:41.768394 6318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:41.777508 6323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.777479 6324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.777570 6326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.777719 6318 server_base.cc:1034] running on GCE node
I20260928 02:41:41.778256 6318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:41.778851 6318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:41.780012 6318 hybrid_clock.cc:648] HybridClock initialized: now 1790563301780001 us; error 25 us; skew 500 ppm
Sep 28 02:41:41 dist-test-slave-rqvb krb5kdc[6077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:41.782640 6318 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:41.783478 6318 webserver.cc:460] Webserver started at http://127.2.123.2:46723/ using document root <none> and password file <none>
I20260928 02:41:41.783999 6318 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:41.784061 6318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:41.784248 6318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:41.786278 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "bf597f6f7d424817bfb2da9bc8f0b195"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "f9ac221d1e5bd3e0ca021613b36faa87"
server_key_iv: "babf80fab88772c119ea7b4ca5009ab7"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.786823 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "bf597f6f7d424817bfb2da9bc8f0b195"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "f9ac221d1e5bd3e0ca021613b36faa87"
server_key_iv: "babf80fab88772c119ea7b4ca5009ab7"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.790429 6318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:41:41.792796 6333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.794086 6318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:41.794188 6318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "bf597f6f7d424817bfb2da9bc8f0b195"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "f9ac221d1e5bd3e0ca021613b36faa87"
server_key_iv: "babf80fab88772c119ea7b4ca5009ab7"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.794284 6318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:41.807054 6318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:41.810030 6318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:41.810210 6318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:41.810787 6318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:41.811594 6318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:41.811640 6318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.811733 6318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:41.811758 6318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.820175 6318 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:43821
I20260928 02:41:41.820206 6446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:43821 every 8 connection(s)
I20260928 02:41:41.820999 6318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:41.825515 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 6318
I20260928 02:41:41.825649 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:41.825914 2540 external_mini_cluster.cc:1428] Setting key d38608373471f9cae0283c39994580ad
Sep 28 02:41:41 dist-test-slave-rqvb krb5kdc[6077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:41:41.835443 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.822929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:60179 (local address 127.2.123.62:39921)
0928 02:41:41.823089 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:41.823093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:41.823897 (+ 804us) server_negotiation.cc:407] Connection header received
0928 02:41:41.825383 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:41.825387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:41.825443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:41.825542 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:41.826887 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.827548 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.828731 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.829013 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.829100 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.831915 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:41.831918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:41.831921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:41.831943 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:41.833207 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.833795 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.833798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.833799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.833879 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:41.834297 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:41.834300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:41.834301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:41.834475 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:41.834551 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:41.835204 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 02:41:41.835307 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:41:41.836305 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:41.823277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39921 (local address 127.2.123.2:60179)
0928 02:41:41.823752 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:41.823802 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:41.825129 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:41.825799 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:41.825806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:41.826220 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:41.826730 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.826749 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.827716 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.827719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.828593 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:41.828602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:41.829159 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:41.829163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:41.829356 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:41.830002 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:41.830020 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:41.831743 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:41.833322 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.833328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.833344 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.833684 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.834006 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:41.834010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:41.834012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:41.834195 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:41.834589 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:41.834596 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:41.834895 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 02:41:41.835104 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 02:41:41.835433 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260928 02:41:41.837497 6447 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39921
I20260928 02:41:41.837844 6447 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:41:41.838418 6447 heartbeater.cc:511] Master 127.2.123.62:39921 requested a full tablet report, sending...
I20260928 02:41:41.839599 6122 ts_manager.cc:194] Registered new tserver with Master: bf597f6f7d424817bfb2da9bc8f0b195 (127.2.123.2:43821)
I20260928 02:41:41.840401 6122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:60179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:41.853423 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39921
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:41.961433 6452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:41.961699 6452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:41.961741 6452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:41.965768 6452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:41.965827 6452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:41.965888 6452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:41.971127 6452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:41.972153 6452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:41.973140 6452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:41.982597 6458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.982627 6460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:41.982774 6457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:41.982774 6452 server_base.cc:1034] running on GCE node
I20260928 02:41:41.983218 6452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:41.983827 6452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:41.984999 6452 hybrid_clock.cc:648] HybridClock initialized: now 1790563301984976 us; error 38 us; skew 500 ppm
Sep 28 02:41:41 dist-test-slave-rqvb krb5kdc[6077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:41.987679 6452 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:41.988567 6452 webserver.cc:460] Webserver started at http://127.2.123.3:43701/ using document root <none> and password file <none>
I20260928 02:41:41.989116 6452 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:41.989181 6452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:41.989372 6452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:41.991421 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "2cd98165d0ed49f3bd7abd46f56d8d8c"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "66c02233ad8e88146521e4e11c9e2667"
server_key_iv: "3a7e0b0aca4d74201b59adb4d85318b2"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.991947 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "2cd98165d0ed49f3bd7abd46f56d8d8c"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "66c02233ad8e88146521e4e11c9e2667"
server_key_iv: "3a7e0b0aca4d74201b59adb4d85318b2"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.995493 6452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 02:41:41.998126 6467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:41.999480 6452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:41.999590 6452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "2cd98165d0ed49f3bd7abd46f56d8d8c"
format_stamp: "Formatted at 2026-09-28 02:41:41 on dist-test-slave-rqvb"
server_key: "66c02233ad8e88146521e4e11c9e2667"
server_key_iv: "3a7e0b0aca4d74201b59adb4d85318b2"
server_key_version: "encryptionkey@0"
I20260928 02:41:41.999698 6452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:42.033087 6452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:42.036695 6452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:42.036897 6452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:42.037503 6452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:42.038408 6452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:42.038465 6452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:42.038518 6452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:42.038546 6452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:42.047014 6452 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:36821
I20260928 02:41:42.047070 6580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:36821 every 8 connection(s)
I20260928 02:41:42.047907 6452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:42.049741 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 6452
I20260928 02:41:42.049883 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:42.050221 2540 external_mini_cluster.cc:1428] Setting key 4cea081987a4a23e4f0bcecb36b40c4d
Sep 28 02:41:42 dist-test-slave-rqvb krb5kdc[6077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:42.061280 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.049843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:58225 (local address 127.2.123.62:39921)
0928 02:41:42.050023 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:42.050029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:42.050921 (+ 892us) server_negotiation.cc:407] Connection header received
0928 02:41:42.052357 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:42.052361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:42.052415 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:42.052487 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:42.053662 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.054254 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.055379 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.055599 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.055696 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.058103 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:42.058106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:42.058108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:42.058132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:42.059374 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:42.059845 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:42.059848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:42.059849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:42.059929 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:42.060226 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:42.060228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:42.060230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:42.060408 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:42.060541 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:42.061025 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 02:41:42.061138 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:41:42.062040 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.050283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39921 (local address 127.2.123.3:58225)
0928 02:41:42.050785 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:42.050837 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:42.052134 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:42.052674 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:42.052682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:42.053031 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:42.053508 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:42.053521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.054374 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.054377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:42.055250 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:42.055259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.055741 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.055745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:42.055910 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.056462 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:42.056479 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:42.057913 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:42.059495 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:42.059499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:42.059512 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:42.059751 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:42.060003 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:42.060005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:42.060007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:42.060126 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:42.060540 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:42.060545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:42.060775 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 02:41:42.060926 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:41:42.061204 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":108,"threads_started":1}
I20260928 02:41:42.063346 6581 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39921
I20260928 02:41:42.063581 6581 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:42.064132 6581 heartbeater.cc:511] Master 127.2.123.62:39921 requested a full tablet report, sending...
I20260928 02:41:42.065068 6122 ts_manager.cc:194] Registered new tserver with Master: 2cd98165d0ed49f3bd7abd46f56d8d8c (127.2.123.3:36821)
I20260928 02:41:42.065591 6122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:58225
I20260928 02:41:42.075968 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:41:42 dist-test-slave-rqvb krb5kdc[6077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:41:42 dist-test-slave-rqvb krb5kdc[6077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:42.101637 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.091750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.2.123.1:39489)
0928 02:41:42.092070 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:42.092085 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:42.092111 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:41:42.092162 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:42.092167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:42.092354 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:42.092471 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:42.093142 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.093949 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.094966 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.095358 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.095447 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.098039 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:42.098052 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:42.098071 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:42.098146 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:42.099827 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:42.100229 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:42.100233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:42.100234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:42.100313 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:42.100670 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:42.100673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:42.100674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:42.100924 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:42.101094 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:42.101303 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:41:42.101434 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
Sep 28 02:41:42 dist-test-slave-rqvb krb5kdc[6077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:42.112603 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:42.104696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.2.123.62:39921)
0928 02:41:42.104843 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:42.104847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:42.104861 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:41:42.104905 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:42.104908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:42.104971 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:42.105092 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:42.105692 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.106269 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.107041 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:42.107254 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:42.107340 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:42.109765 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:42.109768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:42.109770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:42.109805 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:42.111183 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:42.111596 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:42.111599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:42.111600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:42.111664 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:42.111864 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:42.111866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:42.111867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:42.112070 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:42.112186 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:42.112355 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:41:42.112445 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 02:41:42.112991 6122 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37228
I20260928 02:41:42.113716 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 6184
I20260928 02:41:42.119462 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 6318
I20260928 02:41:42.124233 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 6452
I20260928 02:41:42.128744 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 6090
2026-09-28T02:41:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2058 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:42 dist-test-slave-rqvb krb5kdc[6593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:42 dist-test-slave-rqvb krb5kdc[6593](info): set up 2 sockets
Sep 28 02:41:42 dist-test-slave-rqvb krb5kdc[6593](info): commencing operation
krb5kdc: starting...
W20260928 02:41:43.090112 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:43 dist-test-slave-rqvb krb5kdc[6593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:43.213824 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:43511
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:43.326294 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:43.326583 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:43.326637 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:43.330984 6606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:43.331060 6606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:43.331079 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:43.331095 6606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:43.331110 6606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:43.336582 6606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:43511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:43.337687 6606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:43.338709 6606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:43.347036 6614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:43.347085 6612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:43.347127 6606 server_base.cc:1034] running on GCE node
W20260928 02:41:43.347036 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:43.347838 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:43.348928 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:43.350062 6606 hybrid_clock.cc:648] HybridClock initialized: now 1790563303350038 us; error 34 us; skew 500 ppm
Sep 28 02:41:43 dist-test-slave-rqvb krb5kdc[6593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:43.361258 6606 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:43.362074 6606 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:43.362149 6606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:43.362329 6606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:43.364391 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "be4988bdb08b48799e0e30d66f34d4c8"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "b74be19a4880f31d7d47d34036281621"
server_key_iv: "0360a8b7c2dd2ede88abe29f2bf32484"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.364919 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "be4988bdb08b48799e0e30d66f34d4c8"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "b74be19a4880f31d7d47d34036281621"
server_key_iv: "0360a8b7c2dd2ede88abe29f2bf32484"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.368608 6606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:41:43.371234 6620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:43.372598 6606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:41:43.372706 6606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "be4988bdb08b48799e0e30d66f34d4c8"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "b74be19a4880f31d7d47d34036281621"
server_key_iv: "0360a8b7c2dd2ede88abe29f2bf32484"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.372820 6606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:43.396250 6606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:43.400077 6606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:43.400282 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:43.408133 6606 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:43511
I20260928 02:41:43.408380 6672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:43511 every 8 connection(s)
I20260928 02:41:43.408792 6606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:43.410876 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 6606
I20260928 02:41:43.410972 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:43.411221 2540 external_mini_cluster.cc:1428] Setting key 9d61cbb062aad937576df96a1c023c0b
I20260928 02:41:43.411698 6673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:43.418556 6673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8: Bootstrap starting.
Sep 28 02:41:43 dist-test-slave-rqvb krb5kdc[6593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:43.420855 6673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:43.421669 6673 log.cc:824] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:43.424031 6673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8: No bootstrap required, opened a new log
I20260928 02:41:43.426316 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:43.412561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.2.123.62:43511)
0928 02:41:43.412958 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:43.412976 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:43.413018 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:41:43.413838 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:43.413862 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:43.414305 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:43.414688 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:43.415418 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.416908 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.417806 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.418514 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.418678 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:43.421252 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:43.421258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:43.421273 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:43.421305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:43.423224 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:43.423633 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:43.423638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:43.423644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:43.423781 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:43.424016 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:43.424019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:43.424021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:43.424554 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:43.424725 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:43.425042 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:41:43.425324 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260928 02:41:43.428296 6673 raft_consensus.cc:348] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4988bdb08b48799e0e30d66f34d4c8" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43511 } }
I20260928 02:41:43.428551 6673 raft_consensus.cc:374] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:43.428611 6673 raft_consensus.cc:729] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4988bdb08b48799e0e30d66f34d4c8, State: Initialized, Role: FOLLOWER
I20260928 02:41:43.429035 6673 consensus_queue.cc:260] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4988bdb08b48799e0e30d66f34d4c8" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43511 } }
I20260928 02:41:43.429154 6673 raft_consensus.cc:388] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:43.429198 6673 raft_consensus.cc:482] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:43.429266 6673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:43.430159 6673 raft_consensus.cc:504] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4988bdb08b48799e0e30d66f34d4c8" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43511 } }
I20260928 02:41:43.430464 6673 leader_election.cc:302] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be4988bdb08b48799e0e30d66f34d4c8; no voters:
I20260928 02:41:43.430716 6673 leader_election.cc:288] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:43.430838 6678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:43.431114 6678 raft_consensus.cc:686] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [term 1 LEADER]: Becoming Leader. State: Replica: be4988bdb08b48799e0e30d66f34d4c8, State: Running, Role: LEADER
I20260928 02:41:43.431437 6678 consensus_queue.cc:237] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4988bdb08b48799e0e30d66f34d4c8" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43511 } }
I20260928 02:41:43.431926 6673 sys_catalog.cc:564] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:43.432528 6678 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be4988bdb08b48799e0e30d66f34d4c8. Latest consensus state: current_term: 1 leader_uuid: "be4988bdb08b48799e0e30d66f34d4c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4988bdb08b48799e0e30d66f34d4c8" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43511 } } }
I20260928 02:41:43.432713 6678 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:43.432976 6679 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be4988bdb08b48799e0e30d66f34d4c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4988bdb08b48799e0e30d66f34d4c8" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43511 } } }
I20260928 02:41:43.433117 6687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:43.433143 6679 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:43.436290 6687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:43.441869 6687 catalog_manager.cc:1347] Generated new cluster ID: 61cc5da768cc4c9ebaa5d9c18c38cc64
I20260928 02:41:43.441942 6687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:43.463141 6687 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:43.464530 6687 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:43.480047 6687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8: Generated new TSK 0
I20260928 02:41:43.480926 6687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:43.521847 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43511
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:43.631202 6699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:43.631491 6699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:43.631565 6699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:43.635874 6699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:43.635949 6699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:43.636037 6699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:43.641387 6699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:43.642494 6699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:43.643507 6699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:43.653024 6705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:43.653013 6707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:43.653036 6704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:43.653282 6699 server_base.cc:1034] running on GCE node
I20260928 02:41:43.653741 6699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:43.654393 6699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:43.655570 6699 hybrid_clock.cc:648] HybridClock initialized: now 1790563303655548 us; error 26 us; skew 500 ppm
Sep 28 02:41:43 dist-test-slave-rqvb krb5kdc[6593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:43.658257 6699 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:43.658915 6699 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:43.658973 6699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:43.659128 6699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:43.661092 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "db445bc02a634efa91ae1cbfec9c69d8"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "5ad917f1ced98131b61946dbcec55572"
server_key_iv: "a644fa2b60df13694fd18d5ac6179d06"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.661588 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "db445bc02a634efa91ae1cbfec9c69d8"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "5ad917f1ced98131b61946dbcec55572"
server_key_iv: "a644fa2b60df13694fd18d5ac6179d06"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.665159 6699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260928 02:41:43.667688 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:43.669032 6699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:43.669143 6699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "db445bc02a634efa91ae1cbfec9c69d8"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "5ad917f1ced98131b61946dbcec55572"
server_key_iv: "a644fa2b60df13694fd18d5ac6179d06"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.669289 6699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:43.681254 6699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:43.684432 6699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:43.684634 6699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:43.685231 6699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:43.686136 6699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:43.686188 6699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:43.686235 6699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:43.686268 6699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:43.695680 6699 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:35083
I20260928 02:41:43.695770 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:35083 every 8 connection(s)
I20260928 02:41:43.696296 6699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:43.698433 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 6699
I20260928 02:41:43.698530 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:43.698861 2540 external_mini_cluster.cc:1428] Setting key 70f33ddbe4f3ab1b9c336cf1e4ef7f58
Sep 28 02:41:43 dist-test-slave-rqvb krb5kdc[6593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:41:43.711283 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:43.698099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:49917 (local address 127.2.123.62:43511)
0928 02:41:43.698317 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:43.698321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:43.699123 (+ 802us) server_negotiation.cc:407] Connection header received
0928 02:41:43.700584 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:43.700587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:43.700632 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:43.700734 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:43.702291 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.703077 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.704700 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.704938 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.705021 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:43.707828 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:43.707831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:43.707834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:43.707861 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:43.709178 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:43.709696 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:43.709699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:43.709700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:43.709784 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:43.710187 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:43.710190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:43.710191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:43.710376 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:43.710461 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:43.710992 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 02:41:43.711118 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 02:41:43.712093 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:43.698431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:43511 (local address 127.2.123.1:49917)
0928 02:41:43.698951 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:43.699008 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:43.700297 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:43.700918 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:43.700927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:43.701414 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:43.702123 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:43.702145 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.703251 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.703254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:43.704555 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:43.704566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.705103 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.705107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:43.705305 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:43.705912 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:43.705933 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:43.707636 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:43.709298 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:43.709305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:43.709321 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:43.709574 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:43.709876 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:43.709879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:43.709881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:43.710080 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:43.710482 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:43.710489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:43.710743 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 02:41:43.710912 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 02:41:43.711257 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260928 02:41:43.713418 6827 heartbeater.cc:347] Connected to a master server at 127.2.123.62:43511
I20260928 02:41:43.713685 6827 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:43.714210 6827 heartbeater.cc:511] Master 127.2.123.62:43511 requested a full tablet report, sending...
I20260928 02:41:43.715866 6636 ts_manager.cc:194] Registered new tserver with Master: db445bc02a634efa91ae1cbfec9c69d8 (127.2.123.1:35083)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:41:43.717047 6636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:49917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:43.747633 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43511
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:43.855429 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:43.855697 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:43.855743 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:43.859809 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:43.859872 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:43.859933 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:43.865249 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:43.866299 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:43.867298 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:43.877482 6840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:43.877560 6838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:43.877460 6837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:43.878149 6832 server_base.cc:1034] running on GCE node
I20260928 02:41:43.878633 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:43.879434 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:43.880630 6832 hybrid_clock.cc:648] HybridClock initialized: now 1790563303880602 us; error 40 us; skew 500 ppm
Sep 28 02:41:43 dist-test-slave-rqvb krb5kdc[6593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:43.883407 6832 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:43.884207 6832 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:43.884270 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:43.884461 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:43.886665 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "e6266a6963c547bcb39ca73d47f08e29"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "21aba7938f925cf24540e3f2d17dd413"
server_key_iv: "72ac218d622f1b918c647b14eaf8455c"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.887224 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "e6266a6963c547bcb39ca73d47f08e29"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "21aba7938f925cf24540e3f2d17dd413"
server_key_iv: "72ac218d622f1b918c647b14eaf8455c"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.891281 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:41:43.893857 6846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:43.895282 6832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:43.895399 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "e6266a6963c547bcb39ca73d47f08e29"
format_stamp: "Formatted at 2026-09-28 02:41:43 on dist-test-slave-rqvb"
server_key: "21aba7938f925cf24540e3f2d17dd413"
server_key_iv: "72ac218d622f1b918c647b14eaf8455c"
server_key_version: "encryptionkey@0"
I20260928 02:41:43.895507 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:43.918457 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:43.921620 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:43.921799 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:43.922427 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:43.923254 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:43.923310 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:43.923364 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:43.923393 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:43.932273 6832 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:38947
I20260928 02:41:43.932296 6959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:38947 every 8 connection(s)
I20260928 02:41:43.932888 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:43.934062 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 6832
I20260928 02:41:43.934165 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:43.934459 2540 external_mini_cluster.cc:1428] Setting key 0b818db9a5b876d86f6ac9d8fb57fe39
Sep 28 02:41:43 dist-test-slave-rqvb krb5kdc[6593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:41:43.948837 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:43.934877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:49873 (local address 127.2.123.62:43511)
0928 02:41:43.935028 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:43.935033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:43.935916 (+ 883us) server_negotiation.cc:407] Connection header received
0928 02:41:43.938005 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:43.938009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:43.938064 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:43.938138 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:43.939736 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.940288 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.942024 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.942296 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.942394 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:43.945289 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:43.945292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:43.945294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:43.945319 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:43.946710 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:43.947245 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:43.947248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:43.947249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:43.947326 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:43.947729 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:43.947733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:43.947734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:43.947931 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:43.948012 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:43.948537 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 02:41:43.948671 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:41:43.949661 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:43.935095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:43511 (local address 127.2.123.2:49873)
0928 02:41:43.935718 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:43.935823 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:43.937704 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:43.938334 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:43.938344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:43.938862 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:43.939562 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:43.939583 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.940436 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.940440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:43.941848 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:43.941862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:43.942473 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:43.942478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:43.942694 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:43.943331 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:43.943354 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:43.945127 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:43.946810 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:43.946815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:43.946828 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:43.947149 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:43.947438 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:43.947441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:43.947443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:43.947621 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:43.948048 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:43.948054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:43.948301 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 02:41:43.948468 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 02:41:43.948799 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20260928 02:41:43.950924 6960 heartbeater.cc:347] Connected to a master server at 127.2.123.62:43511
I20260928 02:41:43.951174 6960 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:43.951628 6960 heartbeater.cc:511] Master 127.2.123.62:43511 requested a full tablet report, sending...
I20260928 02:41:43.952708 6636 ts_manager.cc:194] Registered new tserver with Master: e6266a6963c547bcb39ca73d47f08e29 (127.2.123.2:38947)
I20260928 02:41:43.953280 6636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:49873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:43.966364 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43511
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:44.076579 6965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:44.076828 6965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:44.076886 6965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:44.080972 6965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:44.081046 6965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:44.081107 6965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:44.086417 6965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:44.087446 6965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:44.088420 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:44.098307 6965 server_base.cc:1034] running on GCE node
W20260928 02:41:44.098331 6971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:44.098307 6970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:44.098407 6973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:44.099047 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:44.099761 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:44.100942 6965 hybrid_clock.cc:648] HybridClock initialized: now 1790563304100917 us; error 38 us; skew 500 ppm
Sep 28 02:41:44 dist-test-slave-rqvb krb5kdc[6593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:44.103977 6965 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:44.104749 6965 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:44.104822 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:44.105021 6965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:44.107239 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "eba2132898bb48ba868e82278b0eeb46"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "57f3bcff978e739f569fba1b6395c7eb"
server_key_iv: "c7fced3a9c7cb55909382cdc9351afe0"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.107810 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "eba2132898bb48ba868e82278b0eeb46"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "57f3bcff978e739f569fba1b6395c7eb"
server_key_iv: "c7fced3a9c7cb55909382cdc9351afe0"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.111788 6965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:41:44.114596 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.116120 6965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:44.116245 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "eba2132898bb48ba868e82278b0eeb46"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "57f3bcff978e739f569fba1b6395c7eb"
server_key_iv: "c7fced3a9c7cb55909382cdc9351afe0"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.116369 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:44.130281 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:44.133654 6965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:44.133838 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:44.134490 6965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:44.135378 6965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:44.135425 6965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.135479 6965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:44.135509 6965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.143982 6965 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:34077
I20260928 02:41:44.144021 7092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:34077 every 8 connection(s)
I20260928 02:41:44.144584 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:44.152952 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 6965
I20260928 02:41:44.153086 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:44.153458 2540 external_mini_cluster.cc:1428] Setting key 7dd996d5bda459b57cb5903149bfedc1
Sep 28 02:41:44 dist-test-slave-rqvb krb5kdc[6593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563304, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:44.158602 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.146370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:49895 (local address 127.2.123.62:43511)
0928 02:41:44.146536 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.146540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.147419 (+ 879us) server_negotiation.cc:407] Connection header received
0928 02:41:44.148704 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.148707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.148755 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.148831 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:44.150087 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.150664 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.151877 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.152132 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.152224 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.155070 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:44.155074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:44.155077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:44.155107 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:44.156557 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:44.157076 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:44.157080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:44.157081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:44.157155 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:44.157473 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:44.157476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:44.157478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:44.157669 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:44.157751 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.158307 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.158427 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:41:44.159405 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.146791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:43511 (local address 127.2.123.3:49895)
0928 02:41:44.147254 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.147297 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:44.148476 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:44.149000 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:44.149023 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:44.149389 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:44.149889 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.149902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.150814 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.150817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.151737 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.151746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.152281 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.152284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.152429 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.153217 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:44.153239 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:44.154923 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:44.156675 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:44.156681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:44.156699 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:44.156975 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:44.157242 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:44.157245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:44.157247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:44.157382 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:44.157773 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:44.157780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:44.158079 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 02:41:44.158253 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:41:44.158582 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260928 02:41:44.160610 7093 heartbeater.cc:347] Connected to a master server at 127.2.123.62:43511
I20260928 02:41:44.160845 7093 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:44.161296 7093 heartbeater.cc:511] Master 127.2.123.62:43511 requested a full tablet report, sending...
I20260928 02:41:44.162323 6636 ts_manager.cc:194] Registered new tserver with Master: eba2132898bb48ba868e82278b0eeb46 (127.2.123.3:34077)
I20260928 02:41:44.162896 6636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:49895
I20260928 02:41:44.167672 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:44.177593 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.169693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.2.123.62:43511)
0928 02:41:44.169820 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.169823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.170037 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:41:44.170167 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.170169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.170214 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.170285 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:44.170976 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.171488 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.172452 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.172720 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.172802 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.174178 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:44.174181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:44.174183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:44.174212 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:44.175747 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:44.176217 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:44.176220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:44.176221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:44.176314 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:44.176692 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:44.176694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:44.176696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:44.176943 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:44.177077 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.177330 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.177475 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:41:44.180784 6636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:44.183844 6636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:44.197770 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.192145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:38947 (local address 127.0.0.1:43438)
0928 02:41:44.192942 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.193050 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:44.193187 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:44.193721 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:44.193727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:44.193747 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:44.193902 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.193908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.195078 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.195081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.196236 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.196245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.197327 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.197331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.197419 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.197453 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:41:44.197522 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:41:44.197611 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":32,"thread_start_us":74,"threads_started":1}
I20260928 02:41:44.198102 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.192861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:34077 (local address 127.0.0.1:56282)
0928 02:41:44.193603 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.193615 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:44.193700 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:44.194204 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:44.194207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:44.194225 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:44.194317 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.194325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.195640 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.195643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.196750 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.196759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.197694 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.197698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.197772 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.197793 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:41:44.197884 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:41:44.197953 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":22,"thread_start_us":104,"threads_started":1}
I20260928 02:41:44.198561 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.192342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.2.123.2:38947)
0928 02:41:44.193242 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.193254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.193277 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:41:44.193356 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.193362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.193558 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.193722 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:44.194038 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.194961 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.196364 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.197194 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.198178 (+ 984us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.198329 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.198370 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.198445 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":221,"threads_started":1}
I20260928 02:41:44.198853 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.192436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:35083 (local address 127.0.0.1:35496)
0928 02:41:44.193095 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.193107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:44.193187 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:44.194892 (+ 1705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:44.194894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:44.194918 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:44.195001 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.195007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.196111 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.196114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.197230 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.197239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.198424 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.198428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.198518 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.198539 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:41:44.198615 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:41:44.198740 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":118,"threads_started":1}
I20260928 02:41:44.198875 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.192751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.2.123.1:35083)
0928 02:41:44.194476 (+ 1725us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.194483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.194504 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:41:44.194572 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.194577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.194752 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.194899 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:44.195125 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.195971 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.197339 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.198139 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.198235 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.198365 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.198672 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.198765 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1624,"thread_start_us":86,"threads_started":1}
I20260928 02:41:44.198581 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.193047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.2.123.3:34077)
0928 02:41:44.193469 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.193480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.193784 (+ 304us) server_negotiation.cc:407] Connection header received
0928 02:41:44.193858 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.193863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.194093 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.194236 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:44.194437 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.195497 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.196873 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.197573 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.197673 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.197809 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.198315 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.198426 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":83,"threads_started":1}
I20260928 02:41:44.201736 6894 tablet_service.cc:1467] Processing CreateTablet for tablet b635560d834a4771a584f9226966ecca (DEFAULT_TABLE table=test-table [id=42cd2c6b09bf4eb2a3dbddad30b5f057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:44.201736 7027 tablet_service.cc:1467] Processing CreateTablet for tablet b635560d834a4771a584f9226966ecca (DEFAULT_TABLE table=test-table [id=42cd2c6b09bf4eb2a3dbddad30b5f057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:44.202921 7027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b635560d834a4771a584f9226966ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:44.202917 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b635560d834a4771a584f9226966ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:44.203877 6761 tablet_service.cc:1467] Processing CreateTablet for tablet b635560d834a4771a584f9226966ecca (DEFAULT_TABLE table=test-table [id=42cd2c6b09bf4eb2a3dbddad30b5f057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:44.204902 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b635560d834a4771a584f9226966ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:44.210330 7111 tablet_bootstrap.cc:492] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: Bootstrap starting.
I20260928 02:41:44.211036 7110 tablet_bootstrap.cc:492] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: Bootstrap starting.
I20260928 02:41:44.211382 7112 tablet_bootstrap.cc:492] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8: Bootstrap starting.
I20260928 02:41:44.213259 7111 tablet_bootstrap.cc:654] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:44.213307 7112 tablet_bootstrap.cc:654] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:44.213307 7110 tablet_bootstrap.cc:654] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:44.214241 7112 log.cc:824] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:44.214383 7111 log.cc:824] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:44.214555 7110 log.cc:824] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:44.216133 7112 tablet_bootstrap.cc:492] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8: No bootstrap required, opened a new log
I20260928 02:41:44.216302 7111 tablet_bootstrap.cc:492] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: No bootstrap required, opened a new log
I20260928 02:41:44.216305 7110 tablet_bootstrap.cc:492] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: No bootstrap required, opened a new log
I20260928 02:41:44.216336 7112 ts_tablet_manager.cc:1397] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:41:44.216487 7111 ts_tablet_manager.cc:1397] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:41:44.216507 7110 ts_tablet_manager.cc:1397] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 02:41:44.220305 7112 raft_consensus.cc:348] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.220315 7111 raft_consensus.cc:348] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.220315 7110 raft_consensus.cc:348] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.220559 7112 raft_consensus.cc:374] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:44.220582 7110 raft_consensus.cc:374] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:44.220582 7111 raft_consensus.cc:374] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:44.220607 7112 raft_consensus.cc:729] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db445bc02a634efa91ae1cbfec9c69d8, State: Initialized, Role: FOLLOWER
I20260928 02:41:44.220638 7111 raft_consensus.cc:729] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6266a6963c547bcb39ca73d47f08e29, State: Initialized, Role: FOLLOWER
I20260928 02:41:44.220638 7110 raft_consensus.cc:729] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba2132898bb48ba868e82278b0eeb46, State: Initialized, Role: FOLLOWER
I20260928 02:41:44.221091 7112 consensus_queue.cc:260] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.221107 7110 consensus_queue.cc:260] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.221092 7111 consensus_queue.cc:260] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.221781 6827 heartbeater.cc:503] Master 127.2.123.62:43511 was elected leader, sending a full tablet report...
I20260928 02:41:44.222270 7112 ts_tablet_manager.cc:1428] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:41:44.222819 6960 heartbeater.cc:503] Master 127.2.123.62:43511 was elected leader, sending a full tablet report...
I20260928 02:41:44.222819 7093 heartbeater.cc:503] Master 127.2.123.62:43511 was elected leader, sending a full tablet report...
I20260928 02:41:44.222826 7110 ts_tablet_manager.cc:1428] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260928 02:41:44.224131 7111 ts_tablet_manager.cc:1428] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: Time spent starting tablet: real 0.008s user 0.000s sys 0.005s
I20260928 02:41:44.232419 7116 raft_consensus.cc:482] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:44.232604 7116 raft_consensus.cc:504] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.233913 7116 leader_election.cc:288] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6266a6963c547bcb39ca73d47f08e29 (127.2.123.2:38947), eba2132898bb48ba868e82278b0eeb46 (127.2.123.3:34077)
I20260928 02:41:44.239128 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.234229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:36719 (local address 127.2.123.2:38947)
0928 02:41:44.234368 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.234372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.234578 (+ 206us) server_negotiation.cc:407] Connection header received
0928 02:41:44.234785 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.234789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.234837 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.234917 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:44.235193 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.236024 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.237803 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.238453 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.238556 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.238616 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.238924 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.238999 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:41:44.239128 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.234627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:56545 (local address 127.2.123.3:34077)
0928 02:41:44.234739 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.234743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.234930 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:41:44.235042 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.235045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.235086 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.235144 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:44.235382 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.236098 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.237803 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.238449 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.238556 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.238616 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.238924 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.239019 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:41:44.239194 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.234527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:34077 (local address 127.2.123.1:56545)
0928 02:41:44.234848 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.234857 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:44.234939 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:44.235175 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:44.235177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:44.235193 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:44.235283 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.235287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.236193 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.236196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.237670 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.237680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.238604 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.238608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.238780 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.238817 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:41:44.238895 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:41:44.239033 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":210,"thread_start_us":103,"threads_started":1}
I20260928 02:41:44.239578 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.234161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:38947 (local address 127.2.123.1:36719)
0928 02:41:44.234426 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.234440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:44.234668 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:44.234939 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:44.234943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:44.234956 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:44.235092 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.235097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.236143 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.236147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.237669 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:44.237680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.238604 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.238608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:44.238775 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.238821 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:41:44.238895 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:41:44.239020 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":141,"thread_start_us":96,"threads_started":1}
I20260928 02:41:44.239861 7047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635560d834a4771a584f9226966ecca" candidate_uuid: "db445bc02a634efa91ae1cbfec9c69d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba2132898bb48ba868e82278b0eeb46" is_pre_election: true
I20260928 02:41:44.240232 7047 raft_consensus.cc:2454] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db445bc02a634efa91ae1cbfec9c69d8 in term 0.
I20260928 02:41:44.240195 6914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635560d834a4771a584f9226966ecca" candidate_uuid: "db445bc02a634efa91ae1cbfec9c69d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6266a6963c547bcb39ca73d47f08e29" is_pre_election: true
I20260928 02:41:44.240485 6914 raft_consensus.cc:2454] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db445bc02a634efa91ae1cbfec9c69d8 in term 0.
I20260928 02:41:44.240830 6715 leader_election.cc:302] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db445bc02a634efa91ae1cbfec9c69d8, eba2132898bb48ba868e82278b0eeb46; no voters:
I20260928 02:41:44.241153 7116 raft_consensus.cc:2781] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:44.241245 7116 raft_consensus.cc:482] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:44.241302 7116 raft_consensus.cc:3037] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:44.242431 7116 raft_consensus.cc:504] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.242755 7116 leader_election.cc:288] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [CANDIDATE]: Term 1 election: Requested vote from peers e6266a6963c547bcb39ca73d47f08e29 (127.2.123.2:38947), eba2132898bb48ba868e82278b0eeb46 (127.2.123.3:34077)
I20260928 02:41:44.243115 6914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635560d834a4771a584f9226966ecca" candidate_uuid: "db445bc02a634efa91ae1cbfec9c69d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6266a6963c547bcb39ca73d47f08e29"
I20260928 02:41:44.243139 7047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635560d834a4771a584f9226966ecca" candidate_uuid: "db445bc02a634efa91ae1cbfec9c69d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba2132898bb48ba868e82278b0eeb46"
I20260928 02:41:44.243221 6914 raft_consensus.cc:3037] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:44.243250 7047 raft_consensus.cc:3037] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:44.244247 6914 raft_consensus.cc:2454] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db445bc02a634efa91ae1cbfec9c69d8 in term 1.
I20260928 02:41:44.244292 7047 raft_consensus.cc:2454] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db445bc02a634efa91ae1cbfec9c69d8 in term 1.
I20260928 02:41:44.244557 6715 leader_election.cc:302] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db445bc02a634efa91ae1cbfec9c69d8, e6266a6963c547bcb39ca73d47f08e29; no voters:
I20260928 02:41:44.244743 7116 raft_consensus.cc:2781] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:44.244951 7116 raft_consensus.cc:686] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 1 LEADER]: Becoming Leader. State: Replica: db445bc02a634efa91ae1cbfec9c69d8, State: Running, Role: LEADER
I20260928 02:41:44.245301 7116 consensus_queue.cc:237] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } }
I20260928 02:41:44.248450 6636 catalog_manager.cc:5495] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to db445bc02a634efa91ae1cbfec9c69d8 (127.2.123.1). New cstate: current_term: 1 leader_uuid: "db445bc02a634efa91ae1cbfec9c69d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db445bc02a634efa91ae1cbfec9c69d8" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 35083 } health_report { overall_health: HEALTHY } } }
I20260928 02:41:44.262643 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.258969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.2.123.1:35083)
0928 02:41:44.259121 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.259124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.259138 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:41:44.259220 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.259222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.259274 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.259360 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:44.259689 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.260218 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.261062 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:44.261272 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:44.261351 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:44.261695 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:44.262221 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:44.262315 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.262457 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.262511 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:41:44.268980 7047 raft_consensus.cc:1261] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 1 FOLLOWER]: Refusing update from remote peer db445bc02a634efa91ae1cbfec9c69d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:44.268970 6914 raft_consensus.cc:1261] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 1 FOLLOWER]: Refusing update from remote peer db445bc02a634efa91ae1cbfec9c69d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:44.269690 7121 consensus_queue.cc:1035] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6266a6963c547bcb39ca73d47f08e29" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:44.269884 7116 consensus_queue.cc:1035] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eba2132898bb48ba868e82278b0eeb46" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:44.277418 7125 mvcc.cc:205] Tried to move back new op lower bound from 7334147294277836800 to 7334147294191722496. Current Snapshot: MvccSnapshot[applied={T|T < 7334147294277836800}]
I20260928 02:41:44.286470 6636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:44.286626 6636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:44.289104 6636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be4988bdb08b48799e0e30d66f34d4c8: Sending DeleteTablet for 3 replicas of tablet b635560d834a4771a584f9226966ecca
I20260928 02:41:44.289865 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet b635560d834a4771a584f9226966ecca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:44 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:43438
I20260928 02:41:44.289925 7027 tablet_service.cc:1514] Processing DeleteTablet for tablet b635560d834a4771a584f9226966ecca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:44 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56282
I20260928 02:41:44.290026 6761 tablet_service.cc:1514] Processing DeleteTablet for tablet b635560d834a4771a584f9226966ecca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:44 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:35496
I20260928 02:41:44.290429 7134 tablet_replica.cc:330] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: stopping tablet replica
I20260928 02:41:44.290459 7136 tablet_replica.cc:330] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8: stopping tablet replica
I20260928 02:41:44.290691 7135 tablet_replica.cc:330] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: stopping tablet replica
I20260928 02:41:44.290879 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 6699
I20260928 02:41:44.290764 7136 raft_consensus.cc:2229] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:44.291278 7136 raft_consensus.cc:2258] T b635560d834a4771a584f9226966ecca P db445bc02a634efa91ae1cbfec9c69d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:44.291846 7135 raft_consensus.cc:2229] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:44.292136 7135 raft_consensus.cc:2258] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:44.292560 7134 raft_consensus.cc:2229] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:44.292831 7134 raft_consensus.cc:2258] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:44.294080 7135 ts_tablet_manager.cc:1905] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:44.294394 7134 ts_tablet_manager.cc:1905] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:44.296826 7135 ts_tablet_manager.cc:1918] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:44.296826 7134 ts_tablet_manager.cc:1918] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:44.296927 7135 log.cc:1196] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/b635560d834a4771a584f9226966ecca
I20260928 02:41:44.296938 7134 log.cc:1196] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/b635560d834a4771a584f9226966ecca
W20260928 02:41:44.297001 6623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:35083 (error 108)
I20260928 02:41:44.297236 7135 ts_tablet_manager.cc:1939] T b635560d834a4771a584f9226966ecca P e6266a6963c547bcb39ca73d47f08e29: Deleting consensus metadata
I20260928 02:41:44.297236 7134 ts_tablet_manager.cc:1939] T b635560d834a4771a584f9226966ecca P eba2132898bb48ba868e82278b0eeb46: Deleting consensus metadata
I20260928 02:41:44.298089 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.297784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:35083 (local address 127.0.0.1:35512)
0928 02:41:44.297911 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.298001 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:41:44.298110 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 6832
I20260928 02:41:44.299090 6622 catalog_manager.cc:4841] TS eba2132898bb48ba868e82278b0eeb46 (127.2.123.3:34077): tablet b635560d834a4771a584f9226966ecca (table test-table [id=42cd2c6b09bf4eb2a3dbddad30b5f057]) successfully deleted
W20260928 02:41:44.299676 6623 catalog_manager.cc:4584] TS db445bc02a634efa91ae1cbfec9c69d8 (127.2.123.1:35083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b635560d834a4771a584f9226966ecca: Network error: Client connection negotiation failed: client connection to 127.2.123.1:35083: connect: Connection refused (error 111)
I20260928 02:41:44.302956 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 6965
I20260928 02:41:44.304332 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.304026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:38947 (local address 127.0.0.1:43446)
0928 02:41:44.304197 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.304248 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.2:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 02:41:44.307786 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 6606
2026-09-28T02:41:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2178 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:44Z Disabled control of system clock
I20260928 02:41:44.336637 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:44399
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:44399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:41:44.449826 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:44.450177 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:44.450245 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:44.454289 7141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:44.454353 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:44.454375 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:44.454397 7141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:44.454418 7141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:44.459154 7141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:44399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:44.460222 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:44.461622 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:44.470144 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:44.470149 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:44.470415 7141 server_base.cc:1034] running on GCE node
W20260928 02:41:44.470139 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:44.470983 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:44.472134 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:44.473305 7141 hybrid_clock.cc:648] HybridClock initialized: now 1790563304473279 us; error 36 us; skew 500 ppm
I20260928 02:41:44.475203 7141 webserver.cc:460] Webserver started at http://127.2.123.62:32935/ using document root <none> and password file <none>
I20260928 02:41:44.475791 7141 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:44.475862 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:44.476061 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:44.478094 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "907c5726334445a09e11edfa1a7c3980"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "51b5bfdfac4b95be1ba9db7bc915bb89"
server_key_iv: "6525d904b55a25d3d65f32911c82416c"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.478621 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "907c5726334445a09e11edfa1a7c3980"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "51b5bfdfac4b95be1ba9db7bc915bb89"
server_key_iv: "6525d904b55a25d3d65f32911c82416c"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.482131 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:44.484432 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.485737 7141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:44.485891 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "907c5726334445a09e11edfa1a7c3980"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "51b5bfdfac4b95be1ba9db7bc915bb89"
server_key_iv: "6525d904b55a25d3d65f32911c82416c"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.486027 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:44.490520 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:44.491098 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:44.491250 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:44.497939 7141 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:44399
I20260928 02:41:44.497987 7207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:44399 every 8 connection(s)
I20260928 02:41:44.498935 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:44.502110 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 7141
I20260928 02:41:44.502202 7208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:44.502219 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:44.502564 2540 external_mini_cluster.cc:1428] Setting key 7b9f95f58661bf943183f151e33f91a3
I20260928 02:41:44.508312 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.503916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.2.123.62:44399)
0928 02:41:44.504526 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.504542 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.504578 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:41:44.505384 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.505404 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.505765 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.506148 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:44.507003 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:44.507008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:44.507020 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:44.507027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:44.507074 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:44.507089 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:44.507201 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.507317 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.507629 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260928 02:41:44.509303 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980: Bootstrap starting.
I20260928 02:41:44.511656 7208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:44.512362 7208 log.cc:824] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:44.514408 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980: No bootstrap required, opened a new log
I20260928 02:41:44.517891 7208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907c5726334445a09e11edfa1a7c3980" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44399 } }
I20260928 02:41:44.518146 7208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:44.518199 7208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907c5726334445a09e11edfa1a7c3980, State: Initialized, Role: FOLLOWER
I20260928 02:41:44.518620 7208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907c5726334445a09e11edfa1a7c3980" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44399 } }
I20260928 02:41:44.518766 7208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:44.518836 7208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:44.518946 7208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:44.519804 7208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907c5726334445a09e11edfa1a7c3980" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44399 } }
I20260928 02:41:44.520143 7208 leader_election.cc:302] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907c5726334445a09e11edfa1a7c3980; no voters:
I20260928 02:41:44.520430 7208 leader_election.cc:288] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:44.520542 7213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:44.520769 7213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [term 1 LEADER]: Becoming Leader. State: Replica: 907c5726334445a09e11edfa1a7c3980, State: Running, Role: LEADER
I20260928 02:41:44.521059 7213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907c5726334445a09e11edfa1a7c3980" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44399 } }
I20260928 02:41:44.521620 7208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:44.522575 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907c5726334445a09e11edfa1a7c3980. Latest consensus state: current_term: 1 leader_uuid: "907c5726334445a09e11edfa1a7c3980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907c5726334445a09e11edfa1a7c3980" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44399 } } }
I20260928 02:41:44.522785 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:44.523054 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907c5726334445a09e11edfa1a7c3980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907c5726334445a09e11edfa1a7c3980" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44399 } } }
I20260928 02:41:44.523150 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:44.523298 7222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:44.526386 7222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:44.531824 7222 catalog_manager.cc:1347] Generated new cluster ID: 9a0b2280a72249bd92a2b8d9d31f6756
I20260928 02:41:44.531893 7222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:44.539796 7222 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:44.540118 7222 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:44.554941 7222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980: Generated new TSK 0
I20260928 02:41:44.555673 7222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:41:44.561510 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:44399
--builtin_ntp_servers=127.2.123.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:41:44.683198 7232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:44.683482 7232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:44.683540 7232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:44.687597 7232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:44.687655 7232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:44.687734 7232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:44.692734 7232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:44.693828 7232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:44.695264 7232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:44.705150 7237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:44.705308 7240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:44.705250 7238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:44.705420 7232 server_base.cc:1034] running on GCE node
I20260928 02:41:44.705856 7232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:44.706543 7232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:44.707726 7232 hybrid_clock.cc:648] HybridClock initialized: now 1790563304707704 us; error 36 us; skew 500 ppm
I20260928 02:41:44.709597 7232 webserver.cc:460] Webserver started at http://127.2.123.1:36885/ using document root <none> and password file <none>
I20260928 02:41:44.710191 7232 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:44.710254 7232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:44.710417 7232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:44.712404 7232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "b07f1aeeabc4433e9a0bcd522523ac29"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "1e539c09c57ad100e397c9cff6ab2c10"
server_key_iv: "03f6ff106f26e37060c1c9ad5c42ed30"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.712896 7232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "b07f1aeeabc4433e9a0bcd522523ac29"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "1e539c09c57ad100e397c9cff6ab2c10"
server_key_iv: "03f6ff106f26e37060c1c9ad5c42ed30"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.716548 7232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:41:44.719012 7246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.720430 7232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:44.720546 7232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "b07f1aeeabc4433e9a0bcd522523ac29"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "1e539c09c57ad100e397c9cff6ab2c10"
server_key_iv: "03f6ff106f26e37060c1c9ad5c42ed30"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.720649 7232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:44.725416 7232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:44.726029 7232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:44.726186 7232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:44.726701 7232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:44.727604 7232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:44.727649 7232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.727699 7232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:44.727728 7232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.736070 7232 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:33995
I20260928 02:41:44.736096 7359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:33995 every 8 connection(s)
I20260928 02:41:44.736971 7232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:44.738476 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 7232
I20260928 02:41:44.738583 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:44.738905 2540 external_mini_cluster.cc:1428] Setting key 3479b623ef50fb2ac9bde3e5dc81063a
I20260928 02:41:44.740876 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:44399
--builtin_ntp_servers=127.2.123.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:41:44.743286 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.738895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:60609 (local address 127.2.123.62:44399)
0928 02:41:44.739048 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.739052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.740316 (+ 1264us) server_negotiation.cc:407] Connection header received
0928 02:41:44.741658 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.741662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.741710 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.741783 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:44.742875 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:44.742879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:44.742882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:44.742893 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:44.742913 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:44.742921 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:44.742994 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.743113 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.743156 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:41:44.744153 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.739380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:44399 (local address 127.2.123.1:60609)
0928 02:41:44.740113 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.740167 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:44.741457 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:44.741981 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:44.741991 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:44.742097 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:44.742620 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:41:44.742621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:44.742665 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:41:44.742737 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:44.742987 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:44.742994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:44.743000 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:41:44.743150 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:41:44.743405 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260928 02:41:44.745561 7360 heartbeater.cc:347] Connected to a master server at 127.2.123.62:44399
I20260928 02:41:44.745814 7360 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:44.746264 7360 heartbeater.cc:511] Master 127.2.123.62:44399 requested a full tablet report, sending...
I20260928 02:41:44.748001 7172 ts_manager.cc:194] Registered new tserver with Master: b07f1aeeabc4433e9a0bcd522523ac29 (127.2.123.1:33995)
W20260928 02:41:44.851608 7363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:44.851867 7363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:44.851904 7363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:44.855903 7363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:44.855970 7363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:44.856033 7363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:44.860666 7363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:44.861649 7363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:44.863014 7363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:44.872870 7363 server_base.cc:1034] running on GCE node
W20260928 02:41:44.872831 7368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:44.872835 7369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:44.873055 7371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:44.873448 7363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:44.874162 7363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:44.875327 7363 hybrid_clock.cc:648] HybridClock initialized: now 1790563304875299 us; error 45 us; skew 500 ppm
I20260928 02:41:44.877214 7363 webserver.cc:460] Webserver started at http://127.2.123.2:36993/ using document root <none> and password file <none>
I20260928 02:41:44.877817 7363 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:44.877892 7363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:44.878149 7363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:44.880317 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "2f11a873546b337946b4b8fae7ae5c36"
server_key_iv: "73c14c8481faef7e62c9169c9d101ce7"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.880839 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "2f11a873546b337946b4b8fae7ae5c36"
server_key_iv: "73c14c8481faef7e62c9169c9d101ce7"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.884605 7363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:41:44.887436 7377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.888831 7363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:41:44.888955 7363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f"
format_stamp: "Formatted at 2026-09-28 02:41:44 on dist-test-slave-rqvb"
server_key: "2f11a873546b337946b4b8fae7ae5c36"
server_key_iv: "73c14c8481faef7e62c9169c9d101ce7"
server_key_version: "encryptionkey@0"
I20260928 02:41:44.889062 7363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:44.893565 7363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:44.894188 7363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:44.894347 7363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:44.894860 7363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:44.895735 7363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:44.895780 7363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.895830 7363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:44.895859 7363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:44.904110 7363 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:34447
I20260928 02:41:44.904206 7490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:34447 every 8 connection(s)
I20260928 02:41:44.904999 7363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:44.907034 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 7363
I20260928 02:41:44.907219 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:44.907478 2540 external_mini_cluster.cc:1428] Setting key 053b82597e4119536c9e92d0cd84761c
I20260928 02:41:44.909616 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:44399
--builtin_ntp_servers=127.2.123.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:41:44.911931 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.906896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:40725 (local address 127.2.123.62:44399)
0928 02:41:44.907047 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:44.907051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:44.908268 (+ 1217us) server_negotiation.cc:407] Connection header received
0928 02:41:44.909872 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:44.909875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:44.909910 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:44.910023 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:44.911357 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:44.911361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:44.911363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:44.911389 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:44.911417 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:44.911429 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:44.911529 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:44.911745 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:41:44.911791 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:41:44.912878 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:44.907367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:44399 (local address 127.2.123.2:40725)
0928 02:41:44.908084 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:44.908136 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:44.909498 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:44.910201 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:44.910211 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:44.910324 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:44.910886 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:41:44.910888 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:44.910948 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:41:44.911130 (+ 182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:44.911576 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:44.911591 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:44.911595 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:41:44.911767 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 02:41:44.911986 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260928 02:41:44.913950 7491 heartbeater.cc:347] Connected to a master server at 127.2.123.62:44399
I20260928 02:41:44.914222 7491 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:44.914709 7491 heartbeater.cc:511] Master 127.2.123.62:44399 requested a full tablet report, sending...
I20260928 02:41:44.915799 7172 ts_manager.cc:194] Registered new tserver with Master: ee2f62bfb49d4ba0809a7e7a1ea2439f (127.2.123.2:34447)
W20260928 02:41:45.018922 7494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:45.019193 7494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:45.019232 7494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:45.023367 7494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:45.023432 7494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:45.023490 7494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:45.028481 7494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:45.029537 7494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:45.030967 7494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:45.040473 7500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:45.040515 7502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:45.040472 7499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:45.040834 7494 server_base.cc:1034] running on GCE node
I20260928 02:41:45.041345 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:45.042033 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:45.043190 7494 hybrid_clock.cc:648] HybridClock initialized: now 1790563305043161 us; error 42 us; skew 500 ppm
I20260928 02:41:45.045504 7494 webserver.cc:460] Webserver started at http://127.2.123.3:39989/ using document root <none> and password file <none>
I20260928 02:41:45.046204 7494 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:45.046276 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:45.046476 7494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:45.048501 7494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "886634b559774edd87340f7bfb4838ad"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "854e39a2acad0203462c045dbd8e3462"
server_key_iv: "68a201db339e1654f302080a62bbe108"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.049041 7494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "886634b559774edd87340f7bfb4838ad"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "854e39a2acad0203462c045dbd8e3462"
server_key_iv: "68a201db339e1654f302080a62bbe108"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.052569 7494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:41:45.054992 7508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:45.056278 7494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:45.056383 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "886634b559774edd87340f7bfb4838ad"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "854e39a2acad0203462c045dbd8e3462"
server_key_iv: "68a201db339e1654f302080a62bbe108"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.056479 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:45.060920 7494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:45.061508 7494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:45.061659 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:45.062244 7494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:45.063153 7494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:45.063197 7494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:45.063246 7494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:45.063271 7494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:45.071444 7494 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:32837
I20260928 02:41:45.071485 7621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:32837 every 8 connection(s)
I20260928 02:41:45.072333 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:45.075795 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 7494
I20260928 02:41:45.075901 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:45.076196 2540 external_mini_cluster.cc:1428] Setting key af641388868728296c062e7797a41e48
I20260928 02:41:45.078310 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.074099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:46121 (local address 127.2.123.62:44399)
0928 02:41:45.074243 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.074248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.075310 (+ 1062us) server_negotiation.cc:407] Connection header received
0928 02:41:45.076632 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.076636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.076677 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.076775 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.077886 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.077890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.077892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.077904 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.077923 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.077931 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.078039 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.078173 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.078211 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:41:45.079174 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.074549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:44399 (local address 127.2.123.3:46121)
0928 02:41:45.075171 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.075212 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:45.076386 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:45.076944 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:45.076954 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:45.077071 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:45.077631 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:41:45.077633 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:45.077673 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:41:45.077750 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:45.078036 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:45.078044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:45.078050 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:41:45.078212 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 02:41:45.078418 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260928 02:41:45.080237 7622 heartbeater.cc:347] Connected to a master server at 127.2.123.62:44399
I20260928 02:41:45.080439 7622 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:45.080811 7622 heartbeater.cc:511] Master 127.2.123.62:44399 requested a full tablet report, sending...
I20260928 02:41:45.081687 7172 ts_manager.cc:194] Registered new tserver with Master: 886634b559774edd87340f7bfb4838ad (127.2.123.3:32837)
I20260928 02:41:45.089874 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:45.093438 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.091848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.2.123.62:44399)
0928 02:41:45.092056 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.092060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.092233 (+ 173us) server_negotiation.cc:407] Connection header received
0928 02:41:45.092335 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.092337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.092370 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.092438 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.093138 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.093141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.093143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.093152 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.093170 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.093178 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.093244 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.093307 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.093338 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:41:45.096283 7172 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:45.099325 7172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:45.111104 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.107804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:32837 (local address 127.0.0.1:46220)
0928 02:41:45.108471 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.108521 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:45.108640 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:45.110035 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:45.110039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:45.110053 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:45.110442 (+ 389us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:41:45.110444 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:45.110454 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:41:45.110478 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:45.110754 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:45.110762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:45.110768 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:41:45.110865 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:41:45.110934 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":81,"threads_started":1}
I20260928 02:41:45.111191 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.108620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.2.123.2:34447)
0928 02:41:45.108999 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.109019 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.109044 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:41:45.109096 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.109101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.109306 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.109441 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.110693 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.110698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.110708 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.110718 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.110762 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.110787 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.110897 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.110942 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.111024 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":96,"threads_started":1}
I20260928 02:41:45.111152 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.107915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:34447 (local address 127.0.0.1:55714)
0928 02:41:45.108710 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.108725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:45.108798 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:45.109434 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:45.109440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:45.109526 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:45.110073 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:41:45.110075 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:45.110097 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:41:45.110166 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:45.110861 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:45.110864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:45.110866 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:41:45.110920 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:41:45.110943 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":105,"threads_started":1}
I20260928 02:41:45.111637 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.108640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.2.123.1:33995)
0928 02:41:45.109058 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.109068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.109237 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:41:45.109293 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.109299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.109462 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.109594 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.110751 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.110756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.110764 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.110776 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.110809 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.110839 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.110926 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.111160 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.111224 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":104,"threads_started":1}
I20260928 02:41:45.111642 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.107929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.2.123.3:32837)
0928 02:41:45.109260 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.109271 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.109288 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:41:45.109329 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.109333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.109462 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.109584 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.110589 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.110595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.110608 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.110619 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.110660 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.110690 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.110784 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.111431 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.111495 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1238,"thread_start_us":156,"threads_started":1}
I20260928 02:41:45.112376 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.108127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:33995 (local address 127.0.0.1:44782)
0928 02:41:45.109068 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.109079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:45.109155 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:45.110247 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:45.110250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:45.110263 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:45.110598 (+ 335us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:41:45.110600 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:45.110609 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:41:45.110632 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:45.110967 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:45.110970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:45.110973 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:41:45.111036 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:41:45.111066 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":87,"threads_started":1}
I20260928 02:41:45.114187 7425 tablet_service.cc:1467] Processing CreateTablet for tablet c6dfce361e0b4b7ca0b3822710c818fe (DEFAULT_TABLE table=test-table [id=6c59171ad1ae478ea80119d0d2fe6abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:45.114264 7556 tablet_service.cc:1467] Processing CreateTablet for tablet c6dfce361e0b4b7ca0b3822710c818fe (DEFAULT_TABLE table=test-table [id=6c59171ad1ae478ea80119d0d2fe6abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:45.115119 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6dfce361e0b4b7ca0b3822710c818fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:45.115253 7556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6dfce361e0b4b7ca0b3822710c818fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:45.115027 7294 tablet_service.cc:1467] Processing CreateTablet for tablet c6dfce361e0b4b7ca0b3822710c818fe (DEFAULT_TABLE table=test-table [id=6c59171ad1ae478ea80119d0d2fe6abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:45.115906 7294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6dfce361e0b4b7ca0b3822710c818fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:45.121001 7639 tablet_bootstrap.cc:492] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Bootstrap starting.
I20260928 02:41:45.123155 7641 tablet_bootstrap.cc:492] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: Bootstrap starting.
I20260928 02:41:45.123664 7639 tablet_bootstrap.cc:654] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:45.124032 7640 tablet_bootstrap.cc:492] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29: Bootstrap starting.
I20260928 02:41:45.124400 7639 log.cc:824] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:45.125828 7640 tablet_bootstrap.cc:654] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:45.126257 7639 tablet_bootstrap.cc:492] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: No bootstrap required, opened a new log
I20260928 02:41:45.126421 7639 ts_tablet_manager.cc:1397] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:41:45.126515 7640 log.cc:824] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:45.128278 7641 tablet_bootstrap.cc:654] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:45.128331 7640 tablet_bootstrap.cc:492] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29: No bootstrap required, opened a new log
I20260928 02:41:45.128515 7640 ts_tablet_manager.cc:1397] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:41:45.129035 7641 log.cc:824] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:45.130241 7639 raft_consensus.cc:348] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.130452 7639 raft_consensus.cc:374] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:45.130501 7639 raft_consensus.cc:729] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2f62bfb49d4ba0809a7e7a1ea2439f, State: Initialized, Role: FOLLOWER
I20260928 02:41:45.130918 7639 consensus_queue.cc:260] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.131102 7641 tablet_bootstrap.cc:492] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: No bootstrap required, opened a new log
I20260928 02:41:45.131285 7641 ts_tablet_manager.cc:1397] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 02:41:45.131613 7491 heartbeater.cc:503] Master 127.2.123.62:44399 was elected leader, sending a full tablet report...
I20260928 02:41:45.131858 7639 ts_tablet_manager.cc:1428] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:41:45.133756 7640 raft_consensus.cc:348] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.134063 7640 raft_consensus.cc:374] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:45.134130 7640 raft_consensus.cc:729] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07f1aeeabc4433e9a0bcd522523ac29, State: Initialized, Role: FOLLOWER
I20260928 02:41:45.134603 7640 consensus_queue.cc:260] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.135257 7360 heartbeater.cc:503] Master 127.2.123.62:44399 was elected leader, sending a full tablet report...
I20260928 02:41:45.135603 7640 ts_tablet_manager.cc:1428] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 02:41:45.136502 7641 raft_consensus.cc:348] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.136706 7641 raft_consensus.cc:374] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:45.136745 7641 raft_consensus.cc:729] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886634b559774edd87340f7bfb4838ad, State: Initialized, Role: FOLLOWER
I20260928 02:41:45.137176 7641 consensus_queue.cc:260] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.137847 7622 heartbeater.cc:503] Master 127.2.123.62:44399 was elected leader, sending a full tablet report...
I20260928 02:41:45.138914 7641 ts_tablet_manager.cc:1428] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260928 02:41:45.156023 7492 tablet.cc:2369] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:45.230096 7646 raft_consensus.cc:482] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:45.230340 7646 raft_consensus.cc:504] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.231614 7646 leader_election.cc:288] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 886634b559774edd87340f7bfb4838ad (127.2.123.3:32837), ee2f62bfb49d4ba0809a7e7a1ea2439f (127.2.123.2:34447)
I20260928 02:41:45.233381 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.231964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:46317 (local address 127.2.123.3:32837)
0928 02:41:45.232082 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.232086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.232102 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:41:45.232215 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.232218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.232261 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.232349 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.233027 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.233031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.233033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.233046 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.233065 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.233074 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.233159 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.233226 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.233261 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:41:45.233381 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.231860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:32837 (local address 127.2.123.1:46317)
0928 02:41:45.231984 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.231999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:45.232065 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:45.232381 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:45.232384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:45.232399 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:45.232876 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:41:45.232878 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:45.232890 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:41:45.232915 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:45.233151 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:45.233154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:45.233157 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:41:45.233226 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:41:45.233260 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:41:45.233649 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.231873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:34447 (local address 127.2.123.1:56397)
0928 02:41:45.232419 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.232432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:45.232509 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:45.232714 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:45.232716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:45.232729 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:45.233120 (+ 391us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:41:45.233121 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:45.233131 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:41:45.233156 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:45.233375 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:45.233378 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:45.233381 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:41:45.233462 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:41:45.233495 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":221,"thread_start_us":89,"threads_started":1}
I20260928 02:41:45.234010 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.231969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:56397 (local address 127.2.123.2:34447)
0928 02:41:45.232082 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.232086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.232505 (+ 419us) server_negotiation.cc:407] Connection header received
0928 02:41:45.232584 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.232587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.232623 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.232698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.233267 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.233271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.233273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.233285 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.233303 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.233310 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.233384 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.233819 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.233860 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:41:45.234107 7576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6dfce361e0b4b7ca0b3822710c818fe" candidate_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886634b559774edd87340f7bfb4838ad" is_pre_election: true
I20260928 02:41:45.234385 7576 raft_consensus.cc:2454] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b07f1aeeabc4433e9a0bcd522523ac29 in term 0.
I20260928 02:41:45.234499 7445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6dfce361e0b4b7ca0b3822710c818fe" candidate_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" is_pre_election: true
I20260928 02:41:45.234751 7445 raft_consensus.cc:2454] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b07f1aeeabc4433e9a0bcd522523ac29 in term 0.
I20260928 02:41:45.234833 7249 leader_election.cc:302] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886634b559774edd87340f7bfb4838ad, b07f1aeeabc4433e9a0bcd522523ac29; no voters:
I20260928 02:41:45.235109 7646 raft_consensus.cc:2781] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:45.235203 7646 raft_consensus.cc:482] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:45.235251 7646 raft_consensus.cc:3037] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:45.236233 7646 raft_consensus.cc:504] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.236557 7646 leader_election.cc:288] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [CANDIDATE]: Term 1 election: Requested vote from peers 886634b559774edd87340f7bfb4838ad (127.2.123.3:32837), ee2f62bfb49d4ba0809a7e7a1ea2439f (127.2.123.2:34447)
I20260928 02:41:45.236876 7445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6dfce361e0b4b7ca0b3822710c818fe" candidate_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f"
I20260928 02:41:45.236999 7445 raft_consensus.cc:3037] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:45.237936 7445 raft_consensus.cc:2454] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b07f1aeeabc4433e9a0bcd522523ac29 in term 1.
W20260928 02:41:45.238073 7361 tablet.cc:2369] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:45.238209 7576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6dfce361e0b4b7ca0b3822710c818fe" candidate_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886634b559774edd87340f7bfb4838ad"
I20260928 02:41:45.238301 7248 leader_election.cc:302] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b07f1aeeabc4433e9a0bcd522523ac29, ee2f62bfb49d4ba0809a7e7a1ea2439f; no voters:
I20260928 02:41:45.238323 7576 raft_consensus.cc:3037] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:45.238458 7646 raft_consensus.cc:2781] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:45.238660 7646 raft_consensus.cc:686] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 1 LEADER]: Becoming Leader. State: Replica: b07f1aeeabc4433e9a0bcd522523ac29, State: Running, Role: LEADER
I20260928 02:41:45.238996 7646 consensus_queue.cc:237] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } }
I20260928 02:41:45.239400 7576 raft_consensus.cc:2454] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b07f1aeeabc4433e9a0bcd522523ac29 in term 1.
I20260928 02:41:45.242298 7172 catalog_manager.cc:5495] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 reported cstate change: term changed from 0 to 1, leader changed from <none> to b07f1aeeabc4433e9a0bcd522523ac29 (127.2.123.1). New cstate: current_term: 1 leader_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b07f1aeeabc4433e9a0bcd522523ac29" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 33995 } health_report { overall_health: HEALTHY } } }
I20260928 02:41:45.268021 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.266575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.2.123.1:33995)
0928 02:41:45.266707 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.266711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.266725 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:41:45.266842 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.266845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.266905 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.266975 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.267649 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.267653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.267656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.267667 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.267686 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.267694 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.267766 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.267845 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.267882 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:41:45.274005 7576 raft_consensus.cc:1261] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 1 FOLLOWER]: Refusing update from remote peer b07f1aeeabc4433e9a0bcd522523ac29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:45.274014 7445 raft_consensus.cc:1261] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 1 FOLLOWER]: Refusing update from remote peer b07f1aeeabc4433e9a0bcd522523ac29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:45.274724 7649 consensus_queue.cc:1035] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2f62bfb49d4ba0809a7e7a1ea2439f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:45.274900 7646 consensus_queue.cc:1035] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886634b559774edd87340f7bfb4838ad" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:45.283761 7654 mvcc.cc:205] Tried to move back new op lower bound from 7334147298395521024 to 7334147298261614592. Current Snapshot: MvccSnapshot[applied={T|T < 7334147298395521024}]
I20260928 02:41:45.290627 7172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:45.290793 7172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:45.293327 7172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 907c5726334445a09e11edfa1a7c3980: Sending DeleteTablet for 3 replicas of tablet c6dfce361e0b4b7ca0b3822710c818fe
I20260928 02:41:45.294143 7425 tablet_service.cc:1514] Processing DeleteTablet for tablet c6dfce361e0b4b7ca0b3822710c818fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:45 UTC) from {username='slave'} at 127.0.0.1:55714
I20260928 02:41:45.294147 7556 tablet_service.cc:1514] Processing DeleteTablet for tablet c6dfce361e0b4b7ca0b3822710c818fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:45 UTC) from {username='slave'} at 127.0.0.1:46220
I20260928 02:41:45.294658 7662 tablet_replica.cc:330] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: stopping tablet replica
I20260928 02:41:45.294865 7663 tablet_replica.cc:330] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: stopping tablet replica
I20260928 02:41:45.294943 7294 tablet_service.cc:1514] Processing DeleteTablet for tablet c6dfce361e0b4b7ca0b3822710c818fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:45 UTC) from {username='slave'} at 127.0.0.1:44782
I20260928 02:41:45.295156 7663 raft_consensus.cc:2229] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:45.295167 7662 raft_consensus.cc:2229] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:45.295511 7663 raft_consensus.cc:2258] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:45.295519 7662 raft_consensus.cc:2258] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:45.295742 7664 tablet_replica.cc:330] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29: stopping tablet replica
I20260928 02:41:45.296075 7664 raft_consensus.cc:2229] T c6dfce361e0b4b7ca0b3822710c818fe P b07f1aeeabc4433e9a0bcd522523ac29 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:45.296178 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 7232
I20260928 02:41:45.298298 7662 ts_tablet_manager.cc:1905] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:45.300621 7662 ts_tablet_manager.cc:1918] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:45.300729 7662 log.cc:1196] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/c6dfce361e0b4b7ca0b3822710c818fe
I20260928 02:41:45.300976 7662 ts_tablet_manager.cc:1939] T c6dfce361e0b4b7ca0b3822710c818fe P ee2f62bfb49d4ba0809a7e7a1ea2439f: Deleting consensus metadata
W20260928 02:41:45.301715 7156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:33995 (error 108)
I20260928 02:41:45.302376 7157 catalog_manager.cc:4841] TS ee2f62bfb49d4ba0809a7e7a1ea2439f (127.2.123.2:34447): tablet c6dfce361e0b4b7ca0b3822710c818fe (table test-table [id=6c59171ad1ae478ea80119d0d2fe6abe]) successfully deleted
I20260928 02:41:45.302853 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.302601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:33995 (local address 127.0.0.1:44810)
0928 02:41:45.302713 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.302779 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:41:45.302855 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 7363
I20260928 02:41:45.303081 7663 ts_tablet_manager.cc:1905] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:41:45.303089 7156 catalog_manager.cc:4584] TS b07f1aeeabc4433e9a0bcd522523ac29 (127.2.123.1:33995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6dfce361e0b4b7ca0b3822710c818fe: Network error: Client connection negotiation failed: client connection to 127.2.123.1:33995: connect: Connection refused (error 111)
I20260928 02:41:45.305564 7663 ts_tablet_manager.cc:1918] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:45.305658 7663 log.cc:1196] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/c6dfce361e0b4b7ca0b3822710c818fe
I20260928 02:41:45.305917 7663 ts_tablet_manager.cc:1939] T c6dfce361e0b4b7ca0b3822710c818fe P 886634b559774edd87340f7bfb4838ad: Deleting consensus metadata
I20260928 02:41:45.307128 7158 catalog_manager.cc:4841] TS 886634b559774edd87340f7bfb4838ad (127.2.123.3:32837): tablet c6dfce361e0b4b7ca0b3822710c818fe (table test-table [id=6c59171ad1ae478ea80119d0d2fe6abe]) successfully deleted
I20260928 02:41:45.308089 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 7494
I20260928 02:41:45.312597 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 7141
2026-09-28T02:41:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1003 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:41:45.332331 2540 mini_oidc.cc:152] Starting JWKS server
I20260928 02:41:45.334235 2540 webserver.cc:460] Webserver started at https://127.0.0.1:36875/ using document root <none> and password file <none>
I20260928 02:41:45.334398 2540 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:41:45.334630 2540 webserver.cc:460] Webserver started at http://127.0.0.1:39923/ using document root <none> and password file <none>
2026-09-28T02:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:45Z Disabled control of system clock
I20260928 02:41:45.343326 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:38265
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:38265
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36875/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/testchainca.cert with env {}
W20260928 02:41:45.454456 7671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:45.454715 7671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:45.454762 7671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:45.458657 7671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:45.458710 7671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:45.458724 7671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:45.458738 7671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:45.458751 7671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:41:45.458765 7671 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:41:45.458777 7671 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36875/jwks.json
I20260928 02:41:45.464026 7671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:38265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36875/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:45.464993 7671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:45.465978 7671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:45.474035 7679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:45.473999 7677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:45.474143 7671 server_base.cc:1034] running on GCE node
W20260928 02:41:45.474009 7676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:45.474817 7671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:45.475924 7671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:45.477102 7671 hybrid_clock.cc:648] HybridClock initialized: now 1790563305477079 us; error 36 us; skew 500 ppm
I20260928 02:41:45.479028 7671 webserver.cc:460] Webserver started at http://127.2.123.62:37257/ using document root <none> and password file <none>
I20260928 02:41:45.479579 7671 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:45.479650 7671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:45.479861 7671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:45.481865 7671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "a9e35f6756501d3aa8061b64a942f26b"
server_key_iv: "f4d83c93a87c1abc27bd88ee48f076ac"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.482450 7671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "a9e35f6756501d3aa8061b64a942f26b"
server_key_iv: "f4d83c93a87c1abc27bd88ee48f076ac"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.486056 7671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:45.488456 7685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:45.489619 7671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:45.489735 7671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "a9e35f6756501d3aa8061b64a942f26b"
server_key_iv: "f4d83c93a87c1abc27bd88ee48f076ac"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.489890 7671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:45.518178 7671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:45.518961 7671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:45.519161 7671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:45.525801 7671 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:38265
I20260928 02:41:45.525858 7737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:38265 every 8 connection(s)
I20260928 02:41:45.526772 7671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:45.528823 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 7671
I20260928 02:41:45.529001 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:45.529322 2540 external_mini_cluster.cc:1428] Setting key 83c9754d7c7a3710822c314e8368d841
I20260928 02:41:45.530369 7738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:45.536048 7738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052: Bootstrap starting.
I20260928 02:41:45.538367 7738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:45.538563 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.530679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.2.123.62:38265)
0928 02:41:45.531351 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.531366 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.531401 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:41:45.532129 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.532148 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.532545 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.532860 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.533286 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.534371 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.535269 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.535794 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.535947 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:45.536781 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.536788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.536806 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.537201 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.537277 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.537298 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.537383 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.537519 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.537780 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260928 02:41:45.539117 7738 log.cc:824] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:45.541039 7738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052: No bootstrap required, opened a new log
I20260928 02:41:45.541306 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:41:45.544549 7738 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 38265 } }
I20260928 02:41:45.544764 7738 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:45.544821 7738 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f0f1e89a7a44b88d6bd0d929c1f052, State: Initialized, Role: FOLLOWER
I20260928 02:41:45.545235 7738 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 38265 } }
I20260928 02:41:45.545361 7738 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:45.545430 7738 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:45.545524 7738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:45.546411 7738 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 38265 } }
I20260928 02:41:45.546707 7738 leader_election.cc:302] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9f0f1e89a7a44b88d6bd0d929c1f052; no voters:
I20260928 02:41:45.546988 7738 leader_election.cc:288] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:45.547068 7743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:45.547259 7743 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [term 1 LEADER]: Becoming Leader. State: Replica: a9f0f1e89a7a44b88d6bd0d929c1f052, State: Running, Role: LEADER
I20260928 02:41:45.547560 7743 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 38265 } }
I20260928 02:41:45.548112 7738 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:45.549054 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f0f1e89a7a44b88d6bd0d929c1f052. Latest consensus state: current_term: 1 leader_uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 38265 } } }
I20260928 02:41:45.549178 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:45.549467 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f0f1e89a7a44b88d6bd0d929c1f052" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 38265 } } }
I20260928 02:41:45.549536 7752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:45.549564 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:45.552445 7752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:45.557745 7752 catalog_manager.cc:1347] Generated new cluster ID: 7428a4dccdda4c5caa605c7fba88f8c3
I20260928 02:41:45.557814 7752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:45.570689 7752 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:45.571939 7752 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:45.580355 7752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9f0f1e89a7a44b88d6bd0d929c1f052: Generated new TSK 0
I20260928 02:41:45.581148 7752 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:41:45.616673 7768 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:41:45.616801 7768 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:41:45.617349 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.600771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.2.123.62:38265)
0928 02:41:45.600884 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.600889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.601107 (+ 218us) server_negotiation.cc:407] Connection header received
0928 02:41:45.601314 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.601317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.601366 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.601456 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:41:45.601701 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.602282 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.603130 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.603346 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.603432 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:45.603755 (+ 323us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:41:45.616942 (+ 13187us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:41:45.617202 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59486: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 02:41:45.617439 7741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59486: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:41:45.618544 7767 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:45.600712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:38265 (local address 127.0.0.1:59486)
0928 02:41:45.601030 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.601041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:45.601163 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:45.601480 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:45.601483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:45.601491 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:41:45.601604 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:45.601609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.602387 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.602391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:45.603025 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:45.603033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.603452 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.603456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:45.603546 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:45.603587 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:41:45.617147 (+ 13560us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:41:45.617373 (+ 226us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.123.62:38265: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.123.62:38265: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:41:45.619130 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 7671
2026-09-28T02:41:45Z chronyd exiting
I20260928 02:41:45.655542 2540 test_util.cc:175] -----------------------------------------------
I20260928 02:41:45.655627 2540 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563278472902-2540-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:41:45.656813 2540 mini_oidc.cc:152] Starting JWKS server
I20260928 02:41:45.657836 2540 webserver.cc:460] Webserver started at https://127.0.0.1:43377/ using document root <none> and password file <none>
I20260928 02:41:45.657933 2540 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:41:45.658210 2540 webserver.cc:460] Webserver started at http://127.0.0.1:45637/ using document root <none> and password file <none>
2026-09-28T02:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:45Z Disabled control of system clock
I20260928 02:41:45.668943 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:44373
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:44373
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43377/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/testchainca.cert with env {}
W20260928 02:41:45.785161 7774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:45.785430 7774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:45.785470 7774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:45.789522 7774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:45.789595 7774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:45.789609 7774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:45.789624 7774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:45.789639 7774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:41:45.789755 7774 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:41:45.789772 7774 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43377/jwks.json
I20260928 02:41:45.795264 7774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:44373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43377/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:45.796281 7774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:45.797237 7774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:45.806057 7774 server_base.cc:1034] running on GCE node
W20260928 02:41:45.805989 7782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:45.805998 7779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:45.805999 7780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:45.806813 7774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:45.807936 7774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:45.809106 7774 hybrid_clock.cc:648] HybridClock initialized: now 1790563305809095 us; error 30 us; skew 500 ppm
I20260928 02:41:45.810900 7774 webserver.cc:460] Webserver started at http://127.2.123.62:36421/ using document root <none> and password file <none>
I20260928 02:41:45.811450 7774 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:45.811515 7774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:45.811712 7774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:45.813704 7774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "e085e07e07c94dea8bc594f662c7a0cf"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "80a6f6dc07e83635cc53057a362ab45c"
server_key_iv: "87be96eedb25562fae09d6ef20640014"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.814303 7774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "e085e07e07c94dea8bc594f662c7a0cf"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "80a6f6dc07e83635cc53057a362ab45c"
server_key_iv: "87be96eedb25562fae09d6ef20640014"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.817803 7774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:41:45.820176 7788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:45.821395 7774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:45.821503 7774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "e085e07e07c94dea8bc594f662c7a0cf"
format_stamp: "Formatted at 2026-09-28 02:41:45 on dist-test-slave-rqvb"
server_key: "80a6f6dc07e83635cc53057a362ab45c"
server_key_iv: "87be96eedb25562fae09d6ef20640014"
server_key_version: "encryptionkey@0"
I20260928 02:41:45.821594 7774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:45.838371 7774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:45.839041 7774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:45.839218 7774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:45.845952 7774 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:44373
I20260928 02:41:45.846032 7840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:44373 every 8 connection(s)
I20260928 02:41:45.846918 7774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:45.850011 7841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:45.854995 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 7774
I20260928 02:41:45.855115 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:45.855370 2540 external_mini_cluster.cc:1428] Setting key aa8cdcf62dc21c1fe6792f501c009e76
I20260928 02:41:45.855975 7841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf: Bootstrap starting.
I20260928 02:41:45.858553 7841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:45.859306 7841 log.cc:824] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:45.861325 7841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf: No bootstrap required, opened a new log
I20260928 02:41:45.864673 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.856720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.2.123.62:44373)
0928 02:41:45.857384 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.857399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.857437 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:41:45.858208 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.858230 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.858710 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.859027 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:45.859496 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.860609 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.861478 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.861948 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.862126 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:45.862970 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:45.862975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:45.862988 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:45.863369 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:45.863422 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:45.863442 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:45.863524 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:45.863668 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:41:45.863935 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20260928 02:41:45.865095 7841 raft_consensus.cc:348] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e085e07e07c94dea8bc594f662c7a0cf" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44373 } }
I20260928 02:41:45.865370 7841 raft_consensus.cc:374] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:45.865417 7841 raft_consensus.cc:729] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e085e07e07c94dea8bc594f662c7a0cf, State: Initialized, Role: FOLLOWER
I20260928 02:41:45.865864 7841 consensus_queue.cc:260] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e085e07e07c94dea8bc594f662c7a0cf" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44373 } }
I20260928 02:41:45.866062 7841 raft_consensus.cc:388] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:45.866135 7841 raft_consensus.cc:482] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:45.866237 7841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:45.867308 7841 raft_consensus.cc:504] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e085e07e07c94dea8bc594f662c7a0cf" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44373 } }
I20260928 02:41:45.867653 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:41:45.867702 7841 leader_election.cc:302] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e085e07e07c94dea8bc594f662c7a0cf; no voters:
I20260928 02:41:45.868098 7841 leader_election.cc:288] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:45.868184 7846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:45.868389 7846 raft_consensus.cc:686] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [term 1 LEADER]: Becoming Leader. State: Replica: e085e07e07c94dea8bc594f662c7a0cf, State: Running, Role: LEADER
I20260928 02:41:45.868738 7846 consensus_queue.cc:237] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e085e07e07c94dea8bc594f662c7a0cf" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44373 } }
I20260928 02:41:45.869436 7841 sys_catalog.cc:564] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:45.870409 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e085e07e07c94dea8bc594f662c7a0cf. Latest consensus state: current_term: 1 leader_uuid: "e085e07e07c94dea8bc594f662c7a0cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e085e07e07c94dea8bc594f662c7a0cf" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44373 } } }
I20260928 02:41:45.870538 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:45.870926 7855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:45.870882 7848 sys_catalog.cc:455] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e085e07e07c94dea8bc594f662c7a0cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e085e07e07c94dea8bc594f662c7a0cf" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 44373 } } }
I20260928 02:41:45.871243 7848 sys_catalog.cc:458] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:45.873907 7855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:45.880424 7855 catalog_manager.cc:1347] Generated new cluster ID: bd72a94a756d49ecb90be287a504226d
I20260928 02:41:45.880496 7855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:45.893890 7855 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:45.895066 7855 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:45.903002 7855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e085e07e07c94dea8bc594f662c7a0cf: Generated new TSK 0
I20260928 02:41:45.903841 7855 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:41:45.933750 7871 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:41:45.933845 7871 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:41:45.934556 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:45.917740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.2.123.62:44373)
0928 02:41:45.917863 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:45.917868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:45.918046 (+ 178us) server_negotiation.cc:407] Connection header received
0928 02:41:45.918212 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:45.918215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:45.918274 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:45.918392 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:41:45.918666 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.919523 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.920427 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.920779 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.920875 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:45.921131 (+ 256us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:41:45.934125 (+ 12994us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:41:45.934413 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:41:45.934664 7870 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:45.917630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:44373 (local address 127.0.0.1:38106)
0928 02:41:45.917932 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:45.917945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:45.918081 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:45.918440 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:45.918443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:45.918452 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:41:45.918554 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:45.918559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.919654 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.919657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:45.920321 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:45.920329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:45.920877 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:45.920880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:45.920941 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:45.920965 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:41:45.934358 (+ 13393us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:41:45.934512 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.123.62:44373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":106,"threads_started":1}
W20260928 02:41:45.934636 7844 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 02:41:45.935294 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 7774
2026-09-28T02:41:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:41:45.973313 2540 mini_oidc.cc:152] Starting JWKS server
I20260928 02:41:45.974229 2540 webserver.cc:460] Webserver started at https://127.0.0.1:43579/ using document root <none> and password file <none>
I20260928 02:41:45.974318 2540 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:41:45.974527 2540 webserver.cc:460] Webserver started at http://127.0.0.1:42125/ using document root <none> and password file <none>
2026-09-28T02:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:45Z Disabled control of system clock
I20260928 02:41:45.982962 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33733
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:34233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:33733
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43579/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:41:46.090570 7877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:46.090837 7877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:46.090873 7877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:46.094847 7877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:46.094907 7877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:46.094920 7877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:46.094936 7877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:46.094949 7877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:41:46.094964 7877 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:41:46.094975 7877 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43579/jwks.json
I20260928 02:41:46.099653 7877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:33733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43579/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:46.100648 7877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:46.101929 7877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:46.110196 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:46.110188 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:46.110236 7877 server_base.cc:1034] running on GCE node
W20260928 02:41:46.110188 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:46.110870 7877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:46.111960 7877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:46.113137 7877 hybrid_clock.cc:648] HybridClock initialized: now 1790563306113100 us; error 48 us; skew 500 ppm
I20260928 02:41:46.115188 7877 webserver.cc:460] Webserver started at http://127.2.123.62:33191/ using document root <none> and password file <none>
I20260928 02:41:46.115729 7877 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:46.115787 7877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:46.115947 7877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:46.118019 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "f48169b6e9a64658b0ddc29d7abec3c6"
format_stamp: "Formatted at 2026-09-28 02:41:46 on dist-test-slave-rqvb"
server_key: "a3014069b2013c37d9a90e36f9abb818"
server_key_iv: "72a71df3639218a556b85460ab437208"
server_key_version: "encryptionkey@0"
I20260928 02:41:46.118556 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "f48169b6e9a64658b0ddc29d7abec3c6"
format_stamp: "Formatted at 2026-09-28 02:41:46 on dist-test-slave-rqvb"
server_key: "a3014069b2013c37d9a90e36f9abb818"
server_key_iv: "72a71df3639218a556b85460ab437208"
server_key_version: "encryptionkey@0"
I20260928 02:41:46.122145 7877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:41:46.124553 7891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:46.125854 7877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:46.125993 7877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "f48169b6e9a64658b0ddc29d7abec3c6"
format_stamp: "Formatted at 2026-09-28 02:41:46 on dist-test-slave-rqvb"
server_key: "a3014069b2013c37d9a90e36f9abb818"
server_key_iv: "72a71df3639218a556b85460ab437208"
server_key_version: "encryptionkey@0"
I20260928 02:41:46.126113 7877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:46.141290 7877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:46.142073 7877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:46.142261 7877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:46.149408 7877 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:33733
I20260928 02:41:46.149546 7943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:33733 every 8 connection(s)
I20260928 02:41:46.150455 7877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:46.153728 7944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:46.158281 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 7877
I20260928 02:41:46.158406 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:46.158706 2540 external_mini_cluster.cc:1428] Setting key 892b6a43982b161df383241cd3819232
I20260928 02:41:46.159436 7944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6: Bootstrap starting.
I20260928 02:41:46.161764 7944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:46.162508 7944 log.cc:824] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:46.164331 7944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6: No bootstrap required, opened a new log
I20260928 02:41:46.167797 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:46.160194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.2.123.62:33733)
0928 02:41:46.160882 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:46.160899 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:46.160936 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:41:46.161569 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:46.161586 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:46.162067 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:46.162387 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:46.162813 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.163777 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.164688 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.165154 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.165297 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:46.166087 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:46.166092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:46.166105 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:46.166491 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:46.166548 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:46.166568 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:46.166653 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:46.166784 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:41:46.167044 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":124,"threads_started":1}
I20260928 02:41:46.167837 7944 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48169b6e9a64658b0ddc29d7abec3c6" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33733 } }
I20260928 02:41:46.168098 7944 raft_consensus.cc:374] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:46.168154 7944 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48169b6e9a64658b0ddc29d7abec3c6, State: Initialized, Role: FOLLOWER
I20260928 02:41:46.168598 7944 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48169b6e9a64658b0ddc29d7abec3c6" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33733 } }
I20260928 02:41:46.168730 7944 raft_consensus.cc:388] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:46.168783 7944 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:46.168866 7944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:46.169939 7944 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48169b6e9a64658b0ddc29d7abec3c6" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33733 } }
I20260928 02:41:46.170315 7944 leader_election.cc:302] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f48169b6e9a64658b0ddc29d7abec3c6; no voters:
I20260928 02:41:46.170655 7944 leader_election.cc:288] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:46.170749 7949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:46.170802 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:41:46.170943 7949 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [term 1 LEADER]: Becoming Leader. State: Replica: f48169b6e9a64658b0ddc29d7abec3c6, State: Running, Role: LEADER
I20260928 02:41:46.171281 7949 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48169b6e9a64658b0ddc29d7abec3c6" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33733 } }
I20260928 02:41:46.171936 7944 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:46.172514 7950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48169b6e9a64658b0ddc29d7abec3c6. Latest consensus state: current_term: 1 leader_uuid: "f48169b6e9a64658b0ddc29d7abec3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48169b6e9a64658b0ddc29d7abec3c6" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33733 } } }
I20260928 02:41:46.172641 7950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:46.172968 7951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f48169b6e9a64658b0ddc29d7abec3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48169b6e9a64658b0ddc29d7abec3c6" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33733 } } }
I20260928 02:41:46.173025 7955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:46.173058 7951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:46.175949 7955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:46.181591 7955 catalog_manager.cc:1347] Generated new cluster ID: 3fa297d9f4c34bc5bcf3e88e5ef0e63b
I20260928 02:41:46.181670 7955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:46.212568 7955 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:46.213521 7955 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:46.218374 7955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48169b6e9a64658b0ddc29d7abec3c6: Generated new TSK 0
I20260928 02:41:46.219086 7955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:41:46.242861 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:46.237562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.2.123.62:33733)
0928 02:41:46.237700 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:46.237704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:46.237879 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:41:46.238062 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:46.238065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:46.238115 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:46.238212 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:41:46.238523 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.239062 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.240306 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.240571 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.240659 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:46.240995 (+ 336us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:41:46.242499 (+ 1504us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:41:46.242721 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35576: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:41:46.242944 7947 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35576: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:41:46.244669 7973 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:46.237468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33733 (local address 127.0.0.1:35576)
0928 02:41:46.237773 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:46.237788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:46.237900 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:46.238230 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:46.238233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:46.238241 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:41:46.238358 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:46.238364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.239204 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.239208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:46.240181 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:46.240191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.240686 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.240690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:46.240779 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:46.240808 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:41:46.242679 (+ 1871us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:41:46.244492 (+ 1813us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.123.62:33733: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":1610,"thread_start_us":102,"threads_started":1}
I20260928 02:41:46.245312 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 7877
E20260928 02:41:46.255373 7974 webserver.cc:533] Webserver: sslize failed:
E20260928 02:41:46.255465 7974 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T02:41:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:46Z Disabled control of system clock
I20260928 02:41:46.292793 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:46369
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:46713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:46369
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/testchainca.cert with env {}
W20260928 02:41:46.399966 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:46.400225 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:46.400264 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:46.404160 7978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:46.404215 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:46.404229 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:46.404244 7978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:46.404258 7978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:41:46.404273 7978 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:41:46.404284 7978 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:41:46.409639 7978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:46369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:46.410660 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:46.411595 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:46.419867 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:46.419914 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:46.419875 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:46.420104 7978 server_base.cc:1034] running on GCE node
I20260928 02:41:46.420608 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:46.421679 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:46.422837 7978 hybrid_clock.cc:648] HybridClock initialized: now 1790563306422817 us; error 33 us; skew 500 ppm
I20260928 02:41:46.424561 7978 webserver.cc:460] Webserver started at http://127.2.123.62:34489/ using document root <none> and password file <none>
I20260928 02:41:46.425073 7978 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:46.425128 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:46.425284 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:46.427223 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "7e25176a27c2467094669405c6a6007e"
format_stamp: "Formatted at 2026-09-28 02:41:46 on dist-test-slave-rqvb"
server_key: "f539d1a4dd5ed86763ee5378a5805c89"
server_key_iv: "adb6683bdd985d7e45582f7e02bdb414"
server_key_version: "encryptionkey@0"
I20260928 02:41:46.427731 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "7e25176a27c2467094669405c6a6007e"
format_stamp: "Formatted at 2026-09-28 02:41:46 on dist-test-slave-rqvb"
server_key: "f539d1a4dd5ed86763ee5378a5805c89"
server_key_iv: "adb6683bdd985d7e45582f7e02bdb414"
server_key_version: "encryptionkey@0"
I20260928 02:41:46.431193 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:41:46.433609 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:46.434967 7978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:46.435065 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "7e25176a27c2467094669405c6a6007e"
format_stamp: "Formatted at 2026-09-28 02:41:46 on dist-test-slave-rqvb"
server_key: "f539d1a4dd5ed86763ee5378a5805c89"
server_key_iv: "adb6683bdd985d7e45582f7e02bdb414"
server_key_version: "encryptionkey@0"
I20260928 02:41:46.435155 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:46.469832 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:46.470559 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:46.470710 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:46.477358 7978 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:46369
I20260928 02:41:46.477386 8044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:46369 every 8 connection(s)
I20260928 02:41:46.478335 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:46.481441 8045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:46.486896 8045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e: Bootstrap starting.
I20260928 02:41:46.488545 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 7978
I20260928 02:41:46.488644 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:46.488890 2540 external_mini_cluster.cc:1428] Setting key df13fb8ef774f24d49c479528faa76a3
I20260928 02:41:46.489423 8045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:46.490335 8045 log.cc:824] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:46.492667 8045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e: No bootstrap required, opened a new log
I20260928 02:41:46.496771 8045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e25176a27c2467094669405c6a6007e" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46369 } }
I20260928 02:41:46.497059 8045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:46.497128 8045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e25176a27c2467094669405c6a6007e, State: Initialized, Role: FOLLOWER
I20260928 02:41:46.497601 8045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e25176a27c2467094669405c6a6007e" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46369 } }
I20260928 02:41:46.497740 8045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:46.497804 8045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:46.497910 8045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:46.498216 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:46.489889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.2.123.62:46369)
0928 02:41:46.490516 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:46.490532 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:46.490568 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:41:46.491351 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:46.491378 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:46.491729 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:46.492022 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:46.492544 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.493556 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.494780 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.495277 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.495403 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:46.496319 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:46.496324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:46.496338 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:41:46.496740 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:46.496798 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:41:46.496817 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:46.496891 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:46.497069 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:41:46.497334 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260928 02:41:46.498886 8045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e25176a27c2467094669405c6a6007e" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46369 } }
I20260928 02:41:46.499213 8045 leader_election.cc:302] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e25176a27c2467094669405c6a6007e; no voters:
I20260928 02:41:46.499501 8045 leader_election.cc:288] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:46.499583 8050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:46.499765 8050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [term 1 LEADER]: Becoming Leader. State: Replica: 7e25176a27c2467094669405c6a6007e, State: Running, Role: LEADER
I20260928 02:41:46.500059 8050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e25176a27c2467094669405c6a6007e" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46369 } }
I20260928 02:41:46.500769 8045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:46.500828 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:41:46.501636 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e25176a27c2467094669405c6a6007e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e25176a27c2467094669405c6a6007e" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46369 } } }
I20260928 02:41:46.501761 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:46.502007 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e25176a27c2467094669405c6a6007e. Latest consensus state: current_term: 1 leader_uuid: "7e25176a27c2467094669405c6a6007e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e25176a27c2467094669405c6a6007e" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46369 } } }
I20260928 02:41:46.502089 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:46.502689 8059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:46.505477 8059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:46.511085 8059 catalog_manager.cc:1347] Generated new cluster ID: db61ae3f02f54c11af36de9fc58a41ab
I20260928 02:41:46.511171 8059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:46.545749 8059 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:46.546778 8059 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:46.554783 8059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e25176a27c2467094669405c6a6007e: Generated new TSK 0
I20260928 02:41:46.555472 8059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:41:47.025302 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:46.566491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.2.123.62:46369)
0928 02:41:46.566610 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:46.566615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:46.566686 (+ 71us) server_negotiation.cc:407] Connection header received
0928 02:41:46.566866 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:46.566869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:46.566924 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:46.567027 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:41:46.567274 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.568073 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.569022 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.569364 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.569463 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:46.569749 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:41:47.024863 (+455114us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:41:47.025136 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57890: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 02:41:47.025388 8048 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57890: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:41:47.025537 8074 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:46.566284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:46369 (local address 127.0.0.1:57890)
0928 02:41:46.566596 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:46.566611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:46.566723 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:46.567060 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:46.567062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:46.567071 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:41:46.567177 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:46.567182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.568205 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.568208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:46.568917 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:46.568926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:46.569488 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:46.569491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:46.569564 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:46.569588 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:41:47.025121 (+455533us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:41:47.025344 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.123.62:46369: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
I20260928 02:41:47.026135 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 7978
2026-09-28T02:41:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (759 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:47 dist-test-slave-rqvb krb5kdc[8086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:47 dist-test-slave-rqvb krb5kdc[8086](info): set up 2 sockets
Sep 28 02:41:47 dist-test-slave-rqvb krb5kdc[8086](info): commencing operation
krb5kdc: starting...
W20260928 02:41:48.230049 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:48 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:48.345528 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40533
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:40533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:48.454241 8099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:48.454512 8099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:48.454552 8099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:48.458477 8099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:48.458535 8099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:48.458554 8099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:48.458570 8099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:48.458583 8099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:48.463728 8099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:40533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:48.464700 8099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:48.465643 8099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:48.474189 8107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:48.474224 8104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:48.474273 8099 server_base.cc:1034] running on GCE node
W20260928 02:41:48.474195 8105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:48.475028 8099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:48.476161 8099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:48.477319 8099 hybrid_clock.cc:648] HybridClock initialized: now 1790563308477301 us; error 33 us; skew 500 ppm
Sep 28 02:41:48 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:48.483222 8099 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:41:48.484211 8099 webserver.cc:460] Webserver started at http://127.2.123.62:41577/ using document root <none> and password file <none>
I20260928 02:41:48.484761 8099 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:48.484863 8099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:48.485075 8099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:48.487223 8099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "7a61aa20bdb74af785e8e5652567bf17"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "72358864e2a353c16f0efac6ee72e153"
server_key_iv: "07df6970cc4a1b93feaab94346d229f4"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.487789 8099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "7a61aa20bdb74af785e8e5652567bf17"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "72358864e2a353c16f0efac6ee72e153"
server_key_iv: "07df6970cc4a1b93feaab94346d229f4"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.491624 8099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 02:41:48.494197 8114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:48.495492 8099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:41:48.495618 8099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "7a61aa20bdb74af785e8e5652567bf17"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "72358864e2a353c16f0efac6ee72e153"
server_key_iv: "07df6970cc4a1b93feaab94346d229f4"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.495710 8099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:48.512424 8099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:48.515578 8099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:48.515751 8099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:48.522457 8099 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:40533
I20260928 02:41:48.522486 8166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:40533 every 8 connection(s)
I20260928 02:41:48.523381 8099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:48.526108 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:48.531394 8167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17: Bootstrap starting.
I20260928 02:41:48.531626 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 8099
I20260928 02:41:48.531731 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:48.531977 2540 external_mini_cluster.cc:1428] Setting key 581fa24ec88979eb4524d0ecc458cb79
I20260928 02:41:48.533859 8167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:48.534731 8167 log.cc:824] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:48.536558 8167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17: No bootstrap required, opened a new log
I20260928 02:41:48.540107 8167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a61aa20bdb74af785e8e5652567bf17" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40533 } }
I20260928 02:41:48.540323 8167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:48.540387 8167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a61aa20bdb74af785e8e5652567bf17, State: Initialized, Role: FOLLOWER
Sep 28 02:41:48 dist-test-slave-rqvb krb5kdc[8086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:48.540805 8167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a61aa20bdb74af785e8e5652567bf17" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40533 } }
I20260928 02:41:48.540937 8167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:48.540997 8167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:48.541090 8167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:48.542164 8167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a61aa20bdb74af785e8e5652567bf17" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40533 } }
I20260928 02:41:48.542526 8167 leader_election.cc:302] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a61aa20bdb74af785e8e5652567bf17; no voters:
I20260928 02:41:48.542850 8167 leader_election.cc:288] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:48.542968 8172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:48.543165 8172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [term 1 LEADER]: Becoming Leader. State: Replica: 7a61aa20bdb74af785e8e5652567bf17, State: Running, Role: LEADER
I20260928 02:41:48.543545 8172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a61aa20bdb74af785e8e5652567bf17" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40533 } }
I20260928 02:41:48.544147 8167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:48.545081 8173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a61aa20bdb74af785e8e5652567bf17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a61aa20bdb74af785e8e5652567bf17" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40533 } } }
I20260928 02:41:48.545203 8173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:48.545502 8174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a61aa20bdb74af785e8e5652567bf17. Latest consensus state: current_term: 1 leader_uuid: "7a61aa20bdb74af785e8e5652567bf17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a61aa20bdb74af785e8e5652567bf17" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40533 } } }
I20260928 02:41:48.545598 8174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:48.546168 8181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:48.546963 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:48.533274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.2.123.62:40533)
0928 02:41:48.533722 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:48.533740 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:48.533781 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:41:48.534610 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:48.534633 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:48.535048 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:48.535404 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:48.536164 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:48.537131 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:48.538048 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:48.538516 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:48.538664 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:48.541181 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:48.541188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:48.541203 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:48.541235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:48.543089 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:48.543583 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:48.543588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:48.543594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:48.543721 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:48.544130 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:48.544133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:48.544134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:48.544677 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:48.544862 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:48.545481 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 02:41:48.545796 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":126,"threads_started":1}
I20260928 02:41:48.549275 8181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:48.554379 8181 catalog_manager.cc:1347] Generated new cluster ID: c2cc3a10b4b648469bb30455f8db57a3
I20260928 02:41:48.554436 8181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:48.563947 8181 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:48.565062 8181 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:48.570029 8181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a61aa20bdb74af785e8e5652567bf17: Generated new TSK 0
I20260928 02:41:48.570780 8181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:41:48.595311 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40533
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:48.703783 8193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:48.704085 8193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:48.704129 8193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:48.708309 8193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:48.708395 8193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:48.708456 8193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:48.713901 8193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:48.714983 8193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:48.715977 8193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:48.725375 8201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:48.725521 8193 server_base.cc:1034] running on GCE node
W20260928 02:41:48.725489 8198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:48.725451 8199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:48.726084 8193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:48.726706 8193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:48.727869 8193 hybrid_clock.cc:648] HybridClock initialized: now 1790563308727852 us; error 34 us; skew 500 ppm
Sep 28 02:41:48 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:48.730535 8193 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:41:48.731350 8193 webserver.cc:460] Webserver started at http://127.2.123.1:41081/ using document root <none> and password file <none>
I20260928 02:41:48.731870 8193 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:48.731932 8193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:48.732117 8193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:48.734117 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "fa4efdbe30854c459a91b997a44aff50"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "3fdfdbf128ca44851885e0656431a412"
server_key_iv: "b9a09a2254857ccd2579cab99b834424"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.734632 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "fa4efdbe30854c459a91b997a44aff50"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "3fdfdbf128ca44851885e0656431a412"
server_key_iv: "b9a09a2254857ccd2579cab99b834424"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.738201 8193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:41:48.740557 8208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:48.741742 8193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:48.741844 8193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "fa4efdbe30854c459a91b997a44aff50"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "3fdfdbf128ca44851885e0656431a412"
server_key_iv: "b9a09a2254857ccd2579cab99b834424"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.741935 8193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:48.756690 8193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:48.759668 8193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:48.759840 8193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:48.760416 8193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:48.761246 8193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:48.761291 8193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:48.761353 8193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:48.761379 8193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:48.770135 8193 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:42357
I20260928 02:41:48.770165 8321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:42357 every 8 connection(s)
I20260928 02:41:48.771070 8193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:41:48 dist-test-slave-rqvb krb5kdc[8086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:48.781262 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 8193
I20260928 02:41:48.781369 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:48.781634 2540 external_mini_cluster.cc:1428] Setting key 15f5f1db02e06eaf32afca4f4e1b8e38
I20260928 02:41:48.785009 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:48.772871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:52931 (local address 127.2.123.62:40533)
0928 02:41:48.773010 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:48.773014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:48.773805 (+ 791us) server_negotiation.cc:407] Connection header received
0928 02:41:48.775265 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:48.775269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:48.775329 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:48.775402 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:48.776563 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:48.777104 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:48.778330 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:48.778594 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:48.778682 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:48.781344 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:48.781348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:48.781351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:48.781385 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:48.782907 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:48.783479 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:48.783484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:48.783485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:48.783561 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:48.783896 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:48.783899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:48.783901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:48.784086 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:48.784170 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:48.784775 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 02:41:48.784887 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:41:48.786135 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:48.773226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40533 (local address 127.2.123.1:52931)
0928 02:41:48.773659 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:48.773701 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:48.775051 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:48.775575 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:48.775583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:48.775947 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:48.776417 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:48.776431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:48.777222 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:48.777225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:48.778206 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:48.778216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:48.778735 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:48.778740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:48.778896 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:48.779439 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:48.779465 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:48.781156 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:48.783019 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:48.783024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:48.783041 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:48.783378 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:48.783653 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:48.783656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:48.783658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:48.783807 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:48.784184 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:48.784190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:48.784468 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 02:41:48.784680 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 02:41:48.785058 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
I20260928 02:41:48.787477 8322 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40533
I20260928 02:41:48.787719 8322 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:48.788201 8322 heartbeater.cc:511] Master 127.2.123.62:40533 requested a full tablet report, sending...
I20260928 02:41:48.789763 8131 ts_manager.cc:194] Registered new tserver with Master: fa4efdbe30854c459a91b997a44aff50 (127.2.123.1:42357)
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:41:48.790915 8131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:52931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:41:48.810107 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40533
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:48.921216 8327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:48.921563 8327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:48.921630 8327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:48.925765 8327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:48.925833 8327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:48.925891 8327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:48.931084 8327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:48.932065 8327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:48.933055 8327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:48.942242 8333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:48.942256 8335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:48.942312 8327 server_base.cc:1034] running on GCE node
W20260928 02:41:48.942250 8332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:48.942829 8327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:48.943466 8327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:48.944620 8327 hybrid_clock.cc:648] HybridClock initialized: now 1790563308944582 us; error 33 us; skew 500 ppm
Sep 28 02:41:48 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:48.947446 8327 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:41:48.948238 8327 webserver.cc:460] Webserver started at http://127.2.123.2:33399/ using document root <none> and password file <none>
I20260928 02:41:48.948751 8327 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:48.948809 8327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:48.948971 8327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:48.950989 8327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "208bb4e67d12463294049c507d124608"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "f6c0fe07f779b1ae7f9ebb37da0471a3"
server_key_iv: "525e52e419dc8e029f185c3eb4ce3456"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.951501 8327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "208bb4e67d12463294049c507d124608"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "f6c0fe07f779b1ae7f9ebb37da0471a3"
server_key_iv: "525e52e419dc8e029f185c3eb4ce3456"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.955400 8327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 02:41:48.957914 8342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:48.959263 8327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:41:48.959373 8327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "208bb4e67d12463294049c507d124608"
format_stamp: "Formatted at 2026-09-28 02:41:48 on dist-test-slave-rqvb"
server_key: "f6c0fe07f779b1ae7f9ebb37da0471a3"
server_key_iv: "525e52e419dc8e029f185c3eb4ce3456"
server_key_version: "encryptionkey@0"
I20260928 02:41:48.959481 8327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:48.980773 8327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:48.983939 8327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:48.984107 8327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:48.984706 8327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:48.985527 8327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:48.985575 8327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:48.985625 8327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:48.985654 8327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:48.994230 8327 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:41987
I20260928 02:41:48.994290 8455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:41987 every 8 connection(s)
I20260928 02:41:48.995136 8327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:48.996521 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 8327
I20260928 02:41:48.996615 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:48.996932 2540 external_mini_cluster.cc:1428] Setting key dcead42ddd539b8455b4911df02e5b89
Sep 28 02:41:48 dist-test-slave-rqvb krb5kdc[8086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:41:49.009480 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:48.996952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:46829 (local address 127.2.123.62:40533)
0928 02:41:48.997094 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:48.997099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:48.998090 (+ 991us) server_negotiation.cc:407] Connection header received
0928 02:41:48.999633 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:48.999637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:48.999705 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:48.999775 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:49.001049 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.001570 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.002767 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.002985 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.003069 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:49.005786 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:49.005790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:49.005792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:49.005818 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:49.007332 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.007914 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.007918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.007920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.008011 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.008412 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.008415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.008417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.008606 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:49.008688 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:49.009182 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 02:41:49.009314 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:41:49.010439 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:48.997393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40533 (local address 127.2.123.2:46829)
0928 02:41:48.997912 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:48.997990 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:48.999401 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:48.999965 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:48.999973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:49.000392 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:49.000901 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:49.000914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.001696 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.001699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:49.002638 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:49.002648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.003129 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.003133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:49.003304 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:49.003911 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:49.003928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:49.005596 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:49.007446 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:49.007452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:49.007464 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:49.007813 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:49.008174 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:49.008177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:49.008179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:49.008323 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:49.008700 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:49.008706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:49.008958 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 02:41:49.009129 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 02:41:49.009458 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":125,"threads_started":1}
I20260928 02:41:49.011859 8456 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40533
I20260928 02:41:49.012104 8456 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:49.012595 8456 heartbeater.cc:511] Master 127.2.123.62:40533 requested a full tablet report, sending...
I20260928 02:41:49.013734 8131 ts_manager.cc:194] Registered new tserver with Master: 208bb4e67d12463294049c507d124608 (127.2.123.2:41987)
I20260928 02:41:49.014578 8131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:46829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:41:49.024888 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40533
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:49.133633 8461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:49.133888 8461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:49.133927 8461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:49.137866 8461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:49.137935 8461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:49.138027 8461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:49.143267 8461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:49.144261 8461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:49.145295 8461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:49.154515 8469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:49.154474 8467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:49.154482 8466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:49.154762 8461 server_base.cc:1034] running on GCE node
I20260928 02:41:49.155189 8461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:49.155823 8461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:49.156981 8461 hybrid_clock.cc:648] HybridClock initialized: now 1790563309156940 us; error 54 us; skew 500 ppm
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:49.159687 8461 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:41:49.160547 8461 webserver.cc:460] Webserver started at http://127.2.123.3:42923/ using document root <none> and password file <none>
I20260928 02:41:49.161041 8461 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:49.161093 8461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:49.161304 8461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:49.163420 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "03efd47a6da841a08616e66fdf936572"
format_stamp: "Formatted at 2026-09-28 02:41:49 on dist-test-slave-rqvb"
server_key: "1b215929e24fc0244f1cecbc92d2f4df"
server_key_iv: "0ba74ff14c25605035718e7f90315ba8"
server_key_version: "encryptionkey@0"
I20260928 02:41:49.163945 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "03efd47a6da841a08616e66fdf936572"
format_stamp: "Formatted at 2026-09-28 02:41:49 on dist-test-slave-rqvb"
server_key: "1b215929e24fc0244f1cecbc92d2f4df"
server_key_iv: "0ba74ff14c25605035718e7f90315ba8"
server_key_version: "encryptionkey@0"
I20260928 02:41:49.167687 8461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:41:49.170270 8476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:49.171577 8461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:41:49.171679 8461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "03efd47a6da841a08616e66fdf936572"
format_stamp: "Formatted at 2026-09-28 02:41:49 on dist-test-slave-rqvb"
server_key: "1b215929e24fc0244f1cecbc92d2f4df"
server_key_iv: "0ba74ff14c25605035718e7f90315ba8"
server_key_version: "encryptionkey@0"
I20260928 02:41:49.171797 8461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:49.184206 8461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:49.187310 8461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:49.187487 8461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:49.188100 8461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:49.188959 8461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:49.189006 8461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:49.189047 8461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:49.189088 8461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:49.197628 8461 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:36719
I20260928 02:41:49.197649 8589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:36719 every 8 connection(s)
I20260928 02:41:49.198688 8461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:49.201051 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 8461
I20260928 02:41:49.201148 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:49.201421 2540 external_mini_cluster.cc:1428] Setting key 310b7303c865ea0e6536c696b8f8def5
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:41:49.212698 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:49.200681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:47799 (local address 127.2.123.62:40533)
0928 02:41:49.200862 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:49.200867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:49.201727 (+ 860us) server_negotiation.cc:407] Connection header received
0928 02:41:49.203119 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:49.203122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:49.203176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:49.203260 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:49.204495 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.205088 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.206264 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.206481 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.206568 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:49.209224 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:49.209228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:49.209230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:49.209255 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:49.210639 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.211181 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.211184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.211186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.211273 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.211669 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.211671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.211672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.211858 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:49.211936 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:49.212453 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 02:41:49.212564 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:41:49.213536 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:49.201067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40533 (local address 127.2.123.3:47799)
0928 02:41:49.201559 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:49.201615 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:49.202925 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:49.203452 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:49.203460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:49.203853 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:49.204358 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:49.204373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.205215 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.205219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:49.206156 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:49.206165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.206646 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.206650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:49.206812 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:49.207413 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:49.207429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:49.209027 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:49.210748 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:49.210754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:49.210767 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:49.211031 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:49.211384 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:49.211387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:49.211389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:49.211567 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:49.211934 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:49.211939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:49.212223 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 02:41:49.212388 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:41:49.212692 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":128,"threads_started":1}
I20260928 02:41:49.214735 8590 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40533
I20260928 02:41:49.214977 8590 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:49.215461 8590 heartbeater.cc:511] Master 127.2.123.62:40533 requested a full tablet report, sending...
I20260928 02:41:49.216500 8130 ts_manager.cc:194] Registered new tserver with Master: 03efd47a6da841a08616e66fdf936572 (127.2.123.3:36719)
I20260928 02:41:49.217067 8130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:47799
I20260928 02:41:49.227070 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/test-admin.keytab.
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:49.242853 2540 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:41:49.245649 8597 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:49.244127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.2.123.1:42357)
0928 02:41:49.244390 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:49.244409 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:49.244437 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:41:49.244494 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:49.244500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:49.244756 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:49.244904 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:49.245472 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58166: BlockingRecv error: recv got EOF from 127.0.0.1:58166 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
W20260928 02:41:49.245635 2540 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:49.246686 2540 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:49.247553 2540 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:49.257200 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:49.248394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.2.123.1:42357)
0928 02:41:49.248506 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:49.248510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:49.248580 (+ 70us) server_negotiation.cc:407] Connection header received
0928 02:41:49.248736 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:49.248740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:49.248785 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:49.248849 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:49.249369 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.250215 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.251040 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.251437 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.251532 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:49.253796 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:49.253805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:49.253819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:49.253844 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:49.255481 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.255891 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.255897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.255898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.255974 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.256230 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.256233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.256235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.256481 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:49.256678 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:49.256893 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 02:41:49.257041 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 02:41:49.261068 8597 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:49.260120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.2.123.1:42357)
0928 02:41:49.260236 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:49.260240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:49.260311 (+ 71us) server_negotiation.cc:407] Connection header received
0928 02:41:49.260428 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:49.260431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:49.260495 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:49.260560 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:49.260960 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58178: BlockingRecv error: recv got EOF from 127.0.0.1:58178 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 02:41:49.261205 2540 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:49.262202 2540 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:49.263127 2540 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:49.271883 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:49.263940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.2.123.1:42357)
0928 02:41:49.264051 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:49.264055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:49.264202 (+ 147us) server_negotiation.cc:407] Connection header received
0928 02:41:49.264285 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:49.264287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:49.264322 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:49.264384 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:49.264898 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.265526 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.266420 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.266659 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.266739 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:49.268873 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:49.268877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:49.268879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:49.268902 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:49.270326 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.270733 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.270736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.270737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.270809 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.271047 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.271050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.271051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.271239 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:49.271345 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:49.271612 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:41:49.271720 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 02:41:49.274662 8597 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:49.273728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.2.123.1:42357)
0928 02:41:49.273837 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:49.273841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:49.273894 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:41:49.274087 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:49.274090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:49.274128 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:49.274193 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:49.274564 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58198: BlockingRecv error: recv got EOF from 127.0.0.1:58198 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:41:49.274778 2540 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563278472902-2540-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:49.275799 2540 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:49.276620 2540 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:41:49.284616 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:49.277388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.2.123.1:42357)
0928 02:41:49.277490 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:49.277494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:49.277582 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:41:49.277710 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:49.277713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:49.277752 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:49.277811 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:49.278324 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.278817 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.279568 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:49.279796 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:49.279879 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:49.281858 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:49.281862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:49.281864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:49.281887 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:49.283152 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.283525 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.283529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.283531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.283612 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:49.283839 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:49.283842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:49.283844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:49.283997 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:49.284156 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:49.284374 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:41:49.284470 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:41:49.285552 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 8193
I20260928 02:41:49.290426 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 8327
I20260928 02:41:49.295271 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 8461
I20260928 02:41:49.299826 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 8099
2026-09-28T02:41:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2034 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8610](info): set up 2 sockets
Sep 28 02:41:49 dist-test-slave-rqvb krb5kdc[8610](info): commencing operation
krb5kdc: starting...
W20260928 02:41:50.258275 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:50 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.62.keytab.
Entry for principal oryx/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.62.keytab.
I20260928 02:41:50.376407 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:42551
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.62.keytab
--principal=oryx/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:42551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:50.482326 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:50.482591 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:50.482646 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:50.486570 8623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:50.486625 8623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:50.486651 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:50.486671 8623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:50.486691 8623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:50.491902 8623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:42551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.62.keytab
--principal=oryx/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:50.492934 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:50.493916 8623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:50.501888 8623 server_base.cc:1034] running on GCE node
W20260928 02:41:50.501823 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:50.501835 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:50.501847 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:50.502694 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:50.503696 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:50.504890 8623 hybrid_clock.cc:648] HybridClock initialized: now 1790563310504875 us; error 29 us; skew 500 ppm
Sep 28 02:41:50 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:50.507634 8623 init.cc:377] Logged in from keytab as oryx/127.2.123.62@KRBTEST.COM (short username oryx)
I20260928 02:41:50.508481 8623 webserver.cc:460] Webserver started at http://127.2.123.62:44213/ using document root <none> and password file <none>
I20260928 02:41:50.508976 8623 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:50.509035 8623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:50.509197 8623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:50.511169 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "4080b99d797d41a09e92d1d747845838"
format_stamp: "Formatted at 2026-09-28 02:41:50 on dist-test-slave-rqvb"
server_key: "60f01462511dc143d2defa798e8601f8"
server_key_iv: "94fdf043010c33c1381c516ca3c271cc"
server_key_version: "encryptionkey@0"
I20260928 02:41:50.511693 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "4080b99d797d41a09e92d1d747845838"
format_stamp: "Formatted at 2026-09-28 02:41:50 on dist-test-slave-rqvb"
server_key: "60f01462511dc143d2defa798e8601f8"
server_key_iv: "94fdf043010c33c1381c516ca3c271cc"
server_key_version: "encryptionkey@0"
I20260928 02:41:50.515233 8623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:41:50.517570 8638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:50.518899 8623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:50.519017 8623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "4080b99d797d41a09e92d1d747845838"
format_stamp: "Formatted at 2026-09-28 02:41:50 on dist-test-slave-rqvb"
server_key: "60f01462511dc143d2defa798e8601f8"
server_key_iv: "94fdf043010c33c1381c516ca3c271cc"
server_key_version: "encryptionkey@0"
I20260928 02:41:50.519125 8623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:50.542867 8623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:50.546005 8623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:50.546176 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:50.552779 8623 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:42551
I20260928 02:41:50.553026 8690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:42551 every 8 connection(s)
I20260928 02:41:50.553718 8623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:50.556396 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:50.561653 8691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: Bootstrap starting.
I20260928 02:41:50.562273 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 8623
I20260928 02:41:50.562381 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:50.562636 2540 external_mini_cluster.cc:1428] Setting key 4ada3e487b37eb69f8f4d053a4ac2bd2
I20260928 02:41:50.564217 8691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:50.564896 8691 log.cc:824] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:50.567152 8691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: No bootstrap required, opened a new log
I20260928 02:41:50.571487 8691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } }
Sep 28 02:41:50 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:50.571719 8691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:50.571781 8691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4080b99d797d41a09e92d1d747845838, State: Initialized, Role: FOLLOWER
I20260928 02:41:50.572192 8691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } }
I20260928 02:41:50.572335 8691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:50.572403 8691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:50.572510 8691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:50.573379 8691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } }
I20260928 02:41:50.573705 8691 leader_election.cc:302] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4080b99d797d41a09e92d1d747845838; no voters:
I20260928 02:41:50.574029 8691 leader_election.cc:288] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:50.574100 8696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:50.574304 8696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 1 LEADER]: Becoming Leader. State: Replica: 4080b99d797d41a09e92d1d747845838, State: Running, Role: LEADER
I20260928 02:41:50.574661 8696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } }
I20260928 02:41:50.575349 8691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:50.575888 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } }
I20260928 02:41:50.576061 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:50.577137 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:50.563860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.2.123.62:42551)
0928 02:41:50.564257 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:50.564271 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:50.564305 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:41:50.564932 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:50.564949 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:50.565333 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:50.565673 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:50.566683 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:50.567696 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:50.568882 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:50.569361 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:50.569479 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:50.572278 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:50.572283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:50.572296 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:50.572323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:50.574007 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:50.574489 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:50.574495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:50.574501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:50.574633 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:50.574938 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:50.574942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:50.574944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:50.575357 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:50.575563 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:50.575852 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:41:50.576162 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260928 02:41:50.578342 8697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4080b99d797d41a09e92d1d747845838. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } }
I20260928 02:41:50.578450 8697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
W20260928 02:41:50.580538 8711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:41:50.580601 8711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:41:50.580686 8701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:50.581549 8701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:50.587218 8701 catalog_manager.cc:1347] Generated new cluster ID: 034e446c90f14499bdd89910a5cde6a9
I20260928 02:41:50.587293 8701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:50.617827 8701 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:50.618803 8701 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:50.626168 8701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: Generated new TSK 0
I20260928 02:41:50.626906 8701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab.
Entry for principal oryx/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab.
I20260928 02:41:50.658459 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42551
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab
--principal=oryx/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:50.796983 8717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:50.797262 8717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:50.797346 8717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:50.801517 8717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:50.801594 8717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:50.801656 8717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:50.806967 8717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab
--principal=oryx/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:50.808007 8717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:50.809041 8717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:50.818296 8723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:50.818298 8722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:50.818284 8725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:50.818471 8717 server_base.cc:1034] running on GCE node
I20260928 02:41:50.818948 8717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:50.819571 8717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:50.820741 8717 hybrid_clock.cc:648] HybridClock initialized: now 1790563310820721 us; error 31 us; skew 500 ppm
Sep 28 02:41:50 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:50.823609 8717 init.cc:377] Logged in from keytab as oryx/127.2.123.1@KRBTEST.COM (short username oryx)
I20260928 02:41:50.824429 8717 webserver.cc:460] Webserver started at http://127.2.123.1:37047/ using document root <none> and password file <none>
I20260928 02:41:50.824987 8717 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:50.825048 8717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:50.825232 8717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:50.827234 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "1ac7e976b8d44a7698eeb189152ba82f"
format_stamp: "Formatted at 2026-09-28 02:41:50 on dist-test-slave-rqvb"
server_key: "39245f4eea1b928a1b05a9d9bc7bd874"
server_key_iv: "2fb4c1a212836cdaf2b1de3b2918e5bf"
server_key_version: "encryptionkey@0"
I20260928 02:41:50.827757 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "1ac7e976b8d44a7698eeb189152ba82f"
format_stamp: "Formatted at 2026-09-28 02:41:50 on dist-test-slave-rqvb"
server_key: "39245f4eea1b928a1b05a9d9bc7bd874"
server_key_iv: "2fb4c1a212836cdaf2b1de3b2918e5bf"
server_key_version: "encryptionkey@0"
I20260928 02:41:50.831329 8717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:41:50.833685 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:50.835054 8717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:50.835161 8717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "1ac7e976b8d44a7698eeb189152ba82f"
format_stamp: "Formatted at 2026-09-28 02:41:50 on dist-test-slave-rqvb"
server_key: "39245f4eea1b928a1b05a9d9bc7bd874"
server_key_iv: "2fb4c1a212836cdaf2b1de3b2918e5bf"
server_key_version: "encryptionkey@0"
I20260928 02:41:50.835260 8717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:50.858781 8717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:50.862294 8717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:50.862497 8717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:50.863096 8717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:50.863914 8717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:50.863956 8717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:50.863998 8717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:50.864027 8717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:50.872453 8717 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:39239
I20260928 02:41:50.872483 8845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:39239 every 8 connection(s)
I20260928 02:41:50.873361 8717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:50.875531 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 8717
I20260928 02:41:50.875640 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:50.875976 2540 external_mini_cluster.cc:1428] Setting key 130e7564c031b8a0312f83f39651f25e
Sep 28 02:41:50 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.2@KRBTEST.COM" created.
I20260928 02:41:50.888963 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:50.875273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:48159 (local address 127.2.123.62:42551)
0928 02:41:50.875476 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:50.875480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:50.876510 (+ 1030us) server_negotiation.cc:407] Connection header received
0928 02:41:50.878253 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:50.878256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:50.878303 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:50.878374 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:50.879831 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:50.880592 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:50.882272 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:50.882507 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:50.882592 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:50.885589 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:50.885591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:50.885594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:50.885618 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:50.886955 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:50.887477 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:50.887480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:50.887481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:50.887556 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:50.887874 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:50.887877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:50.887879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:50.888063 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:50.888146 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:50.888733 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 02:41:50.888839 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 02:41:50.890151 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:50.875789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.1:48159)
0928 02:41:50.876351 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:50.876404 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:50.877914 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:50.878580 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:50.878589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:50.879068 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:50.879676 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:50.879690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:50.880749 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:50.880752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:50.882142 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:50.882154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:50.882677 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:50.882683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:50.882893 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:50.883645 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:50.883668 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:50.885407 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:50.887068 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:50.887074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:50.887091 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:50.887377 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:50.887637 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:50.887639 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:50.887642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:50.887787 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:50.888159 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:50.888166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:50.888459 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 02:41:50.888670 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 02:41:50.889039 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":137,"threads_started":1}
I20260928 02:41:50.891386 8846 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42551
I20260928 02:41:50.891610 8846 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:50.892086 8846 heartbeater.cc:511] Master 127.2.123.62:42551 requested a full tablet report, sending...
I20260928 02:41:50.893713 8655 ts_manager.cc:194] Registered new tserver with Master: 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239)
I20260928 02:41:50.895280 8655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:48159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab.
Entry for principal oryx/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab.
I20260928 02:41:50.904047 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42551
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab
--principal=oryx/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:51.011096 8851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:51.011377 8851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:51.011420 8851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:51.015378 8851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:51.015429 8851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:51.015484 8851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:51.020699 8851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab
--principal=oryx/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:51.021700 8851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:51.022787 8851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:51.032071 8856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:51.032137 8857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:51.032148 8851 server_base.cc:1034] running on GCE node
W20260928 02:41:51.032073 8859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:51.032703 8851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:51.033353 8851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:51.034523 8851 hybrid_clock.cc:648] HybridClock initialized: now 1790563311034504 us; error 33 us; skew 500 ppm
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:51.037127 8851 init.cc:377] Logged in from keytab as oryx/127.2.123.2@KRBTEST.COM (short username oryx)
I20260928 02:41:51.038075 8851 webserver.cc:460] Webserver started at http://127.2.123.2:33547/ using document root <none> and password file <none>
I20260928 02:41:51.038640 8851 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:51.038707 8851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:51.038931 8851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:51.040987 8851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "cdb2f2879ec5457bab7a163cc9cb0059"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "0163c20e640c90ffdb10e08f14f9746f"
server_key_iv: "f8d6bce67447100d73f01da4c5ab13e7"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.041550 8851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "cdb2f2879ec5457bab7a163cc9cb0059"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "0163c20e640c90ffdb10e08f14f9746f"
server_key_iv: "f8d6bce67447100d73f01da4c5ab13e7"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.045513 8851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 02:41:51.048077 8866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.049388 8851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:51.049510 8851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "cdb2f2879ec5457bab7a163cc9cb0059"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "0163c20e640c90ffdb10e08f14f9746f"
server_key_iv: "f8d6bce67447100d73f01da4c5ab13e7"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.049612 8851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:51.075647 8851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:51.078989 8851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:51.079174 8851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:51.079735 8851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:51.080509 8851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:51.080554 8851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.080605 8851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:51.080627 8851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.088706 8851 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:41349
I20260928 02:41:51.088769 8979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:41349 every 8 connection(s)
I20260928 02:41:51.089571 8851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:51.090227 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 8851
I20260928 02:41:51.090341 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:51.090637 2540 external_mini_cluster.cc:1428] Setting key 2b49e8244e26bad5f13acaa53ed35e45
WARNING: no policy specified for oryx/127.2.123.3@KRBTEST.COM; defaulting to no policy
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.3@KRBTEST.COM" created.
I20260928 02:41:51.105414 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.091565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:33097 (local address 127.2.123.62:42551)
0928 02:41:51.091731 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.091735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.092750 (+ 1015us) server_negotiation.cc:407] Connection header received
0928 02:41:51.094459 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.094463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.094521 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.094609 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.096078 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.096668 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.098401 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.098654 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.098746 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.101604 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.101607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.101609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.101634 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.103081 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.103730 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.103733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.103736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.103830 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.104200 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.104203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.104205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.104431 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.104523 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.105102 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.105245 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:41:51.106411 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.092024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.2:33097)
0928 02:41:51.092572 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.092632 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.094240 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.094804 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.094811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.095232 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.095910 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.095931 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.096823 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.096827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.098269 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.098282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.098853 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.098858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.099068 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.099707 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.099732 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.101409 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.103200 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.103207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.103226 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.103625 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.103945 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.103948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.103949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.104103 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.104555 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.104560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.104829 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.105034 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.105426 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":124,"threads_started":1}
I20260928 02:41:51.107769 8980 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42551
I20260928 02:41:51.108013 8980 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:51.108558 8980 heartbeater.cc:511] Master 127.2.123.62:42551 requested a full tablet report, sending...
I20260928 02:41:51.109700 8655 ts_manager.cc:194] Registered new tserver with Master: cdb2f2879ec5457bab7a163cc9cb0059 (127.2.123.2:41349)
I20260928 02:41:51.110375 8655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:33097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab.
Entry for principal oryx/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab.
I20260928 02:41:51.120404 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42551
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab
--principal=oryx/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:51.228205 8985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:51.228478 8985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:51.228521 8985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:51.232503 8985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:51.232570 8985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:51.232627 8985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:51.237883 8985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab
--principal=oryx/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:51.238999 8985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:51.240021 8985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:51.249682 8993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:51.249686 8991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:51.249634 8990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:51.249722 8985 server_base.cc:1034] running on GCE node
I20260928 02:41:51.250365 8985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:51.250988 8985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:51.252173 8985 hybrid_clock.cc:648] HybridClock initialized: now 1790563311252152 us; error 39 us; skew 500 ppm
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:51.254937 8985 init.cc:377] Logged in from keytab as oryx/127.2.123.3@KRBTEST.COM (short username oryx)
I20260928 02:41:51.255877 8985 webserver.cc:460] Webserver started at http://127.2.123.3:37539/ using document root <none> and password file <none>
I20260928 02:41:51.256402 8985 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:51.256465 8985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:51.256665 8985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:51.258718 8985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "79410553cd9a4050b052449aa189151e"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "8c957e34c86a535b0cc9b09e67d02b8b"
server_key_iv: "76c87d1353cd2a316dcfe5617d5469e2"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.259259 8985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "79410553cd9a4050b052449aa189151e"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "8c957e34c86a535b0cc9b09e67d02b8b"
server_key_iv: "76c87d1353cd2a316dcfe5617d5469e2"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.263010 8985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:41:51.265544 9000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.266952 8985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:51.267120 8985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "79410553cd9a4050b052449aa189151e"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "8c957e34c86a535b0cc9b09e67d02b8b"
server_key_iv: "76c87d1353cd2a316dcfe5617d5469e2"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.267225 8985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:51.293087 8985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:51.296448 8985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:51.296638 8985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:51.297255 8985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:51.298125 8985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:51.298172 8985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.298223 8985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:51.298250 8985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.306705 8985 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:35061
I20260928 02:41:51.306731 9113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:35061 every 8 connection(s)
I20260928 02:41:51.307588 8985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:51.316870 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 8985
I20260928 02:41:51.316973 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:51.317256 2540 external_mini_cluster.cc:1428] Setting key a6bf541ee240797126e39ab44dfa01a1
I20260928 02:41:51.321622 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.309460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:42943 (local address 127.2.123.62:42551)
0928 02:41:51.309673 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.309678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.310545 (+ 867us) server_negotiation.cc:407] Connection header received
0928 02:41:51.311834 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.311838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.311886 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.311958 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.313188 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.313730 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.314922 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.315161 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.315255 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.317862 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.317866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.317869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.317897 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.319456 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.320096 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.320099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.320100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.320193 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.320521 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.320524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.320525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.320726 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.320810 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.321373 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.321486 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 02:41:51.322540 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.309876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.3:42943)
0928 02:41:51.310367 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.310412 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.311624 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.312131 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.312139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.312529 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.313030 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.313044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.313852 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.313855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.314800 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.314809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.315339 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.315343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.315493 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.316065 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.316082 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.317712 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.319609 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.319616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.319633 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.319929 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.320273 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.320275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.320276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.320416 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.320839 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.320845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.321137 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.321310 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.321614 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":116,"threads_started":1}
I20260928 02:41:51.323817 9114 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42551
I20260928 02:41:51.324071 9114 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:51.324592 9114 heartbeater.cc:511] Master 127.2.123.62:42551 requested a full tablet report, sending...
I20260928 02:41:51.325650 8655 ts_manager.cc:194] Registered new tserver with Master: 79410553cd9a4050b052449aa189151e (127.2.123.3:35061)
I20260928 02:41:51.326319 8655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:42943
I20260928 02:41:51.331588 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.123.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.61.keytab.
Entry for principal oryx/127.2.123.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.61.keytab.
I20260928 02:41:51.357388 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.61:40139
--webserver_interface=127.2.123.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.61.keytab
--principal=oryx/127.2.123.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:42551,127.2.123.61:40139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:51.465718 9119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:51.466022 9119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:51.466090 9119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:51.470062 9119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:51.470126 9119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:51.470152 9119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:51.470182 9119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:51.470211 9119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:51.475589 9119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:42551,127.2.123.61:40139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.61.keytab
--principal=oryx/127.2.123.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.61:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.123.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
W20260928 02:41:51.475801 9119 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.123.62:42551,127.2.123.61:40139'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:41:51.476722 9119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:51.477730 9119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:51.485898 9125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:51.486079 9119 server_base.cc:1034] running on GCE node
W20260928 02:41:51.485901 9127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:51.485918 9124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:51.486722 9119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:51.487821 9119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:51.488977 9119 hybrid_clock.cc:648] HybridClock initialized: now 1790563311488939 us; error 54 us; skew 500 ppm
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:51.491782 9119 init.cc:377] Logged in from keytab as oryx/127.2.123.61@KRBTEST.COM (short username oryx)
I20260928 02:41:51.492745 9119 webserver.cc:460] Webserver started at http://127.2.123.61:38137/ using document root <none> and password file <none>
I20260928 02:41:51.493321 9119 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:51.493387 9119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:51.493580 9119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:51.495626 9119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data/instance:
uuid: "02f615b6adfd4a0eace185decbe5f9f6"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "db23702846b1ae7a364cfb67fba999cf"
server_key_iv: "3cf69a408020f87cae782d724c7904a9"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.496162 9119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal/instance:
uuid: "02f615b6adfd4a0eace185decbe5f9f6"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "db23702846b1ae7a364cfb67fba999cf"
server_key_iv: "3cf69a408020f87cae782d724c7904a9"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.499950 9119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:41:51.502453 9134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.503775 9119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:51.503888 9119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
uuid: "02f615b6adfd4a0eace185decbe5f9f6"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "db23702846b1ae7a364cfb67fba999cf"
server_key_iv: "3cf69a408020f87cae782d724c7904a9"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.503994 9119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:51.527326 9119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:51.530149 9119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:51.530318 9119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:51.536738 9119 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.61:40139
I20260928 02:41:51.536980 9186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.61:40139 every 8 connection(s)
I20260928 02:41:51.537663 9119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
I20260928 02:41:51.540376 9187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:51.543464 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 9119
I20260928 02:41:51.544035 9187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.61@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.123.61@KRBTEST.COM
I20260928 02:41:51.556722 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.545873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.2.123.62:42551)
0928 02:41:51.546017 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.546021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.546291 (+ 270us) server_negotiation.cc:407] Connection header received
0928 02:41:51.546628 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.546631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.546683 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.546748 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.547511 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.548325 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.549661 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.550073 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.550178 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.552915 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.552919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.552922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.552950 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.554525 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.555104 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.555109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.555110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.555196 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.555550 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.555553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.555554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.555749 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.555840 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.556430 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.556556 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:41:51.557618 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.545811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:59856)
0928 02:41:51.546171 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.546207 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.546463 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.546807 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.546812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.547142 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.547383 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.547396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.548445 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.548448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.549527 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.549537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.550221 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.550224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.550416 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.551078 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.551098 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.552731 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.554631 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.554639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.554649 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.555004 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.555284 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.555287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.555289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.555460 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.555859 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.555865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.556137 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.556372 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.556691 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":45,"thread_start_us":140,"threads_started":1}
I20260928 02:41:51.557680 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.544916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.2.123.61:40139)
0928 02:41:51.545392 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.545406 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.545439 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:41:51.546294 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.546316 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.546578 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.546749 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.547562 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.548678 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.549757 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.550385 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.550489 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.553490 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.553497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.553512 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.553536 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.555194 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.555928 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.555934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.555936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.556022 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.556313 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.556316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.556317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.556583 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.556755 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.557376 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.557502 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":339,"thread_start_us":143,"threads_started":1}
I20260928 02:41:51.559000 9187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.61@KRBTEST.COM for oryx/127.2.123.61@KRBTEST.COM
I20260928 02:41:51.566701 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.559463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.0.0.1:38026)
0928 02:41:51.559643 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.559655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.559722 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.560071 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.560074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.560209 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.560286 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.560290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.560948 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.560951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.561447 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.561453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.562330 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.562333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.562417 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.562832 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.562842 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.564082 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.565442 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.565445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.565447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.565620 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.565849 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.565852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.565854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.565920 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.566248 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.566251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.566327 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.566406 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.566538 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 02:41:51.566701 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.559819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.2.123.61:40139)
0928 02:41:51.559918 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.559921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.559929 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:41:51.559951 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.559953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.559981 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.560046 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.560378 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.560860 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.561540 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.562236 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.562325 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.564191 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.564195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.564197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.564218 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.565351 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.565698 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.565701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.565703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.565776 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.566011 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.566014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.566016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.566170 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.566248 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.566456 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.566589 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":42}
I20260928 02:41:51.569298 9187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Bootstrap starting.
I20260928 02:41:51.571580 9187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:51.572220 9187 log.cc:824] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:51.574039 9187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: No bootstrap required, opened a new log
I20260928 02:41:51.577052 9187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } }
I20260928 02:41:51.577253 9187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:51.577296 9187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f615b6adfd4a0eace185decbe5f9f6, State: Initialized, Role: FOLLOWER
I20260928 02:41:51.577795 9187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } }
I20260928 02:41:51.578260 9193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } } }
I20260928 02:41:51.578377 9193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:51.578960 9187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:41:51.583074 9204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:41:51.583142 9204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:41:51.585139 9119 master_runner.cc:428] Detected that this master 02f615b6adfd4a0eace185decbe5f9f6 is joining an existing cluster
I20260928 02:41:51.585186 9119 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.123.61:40139
I20260928 02:41:51.594916 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.587490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.2.123.61:40139)
0928 02:41:51.587589 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.587593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.587851 (+ 258us) server_negotiation.cc:407] Connection header received
0928 02:41:51.587960 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.587963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.588007 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.588086 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.588495 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.588953 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.590012 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.590368 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.590472 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.591972 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.591975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.591978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.592006 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.593341 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.593700 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.593703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.593705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.593800 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.594108 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.594111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.594113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.594300 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.594364 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.594651 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.594753 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:41:51.595098 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.587050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:59872)
0928 02:41:51.587369 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.587382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.587519 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.587878 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.587881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.588077 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.588213 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.588219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.589267 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.589270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.590112 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.590119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.590633 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.590636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.590711 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.591243 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.591259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.591743 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.593690 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.593693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.593695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.593910 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.594181 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.594184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.594186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.594260 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.594600 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.594603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.594685 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.594766 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.594892 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260928 02:41:51.595335 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.587118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.2.123.62:42551)
0928 02:41:51.587225 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.587228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.587447 (+ 219us) server_negotiation.cc:407] Connection header received
0928 02:41:51.587631 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.587634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.587685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.587738 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.588328 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.589171 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.590206 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.590429 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.590520 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.591891 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.591895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.591898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.591925 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.593595 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.594019 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.594022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.594024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.594103 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.594335 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.594338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.594340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.594517 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.594598 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.594825 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.594930 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:41:51.595455 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.587538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.0.0.1:38032)
0928 02:41:51.587757 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.587770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.587852 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.588103 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.588106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.588286 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.588400 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.588405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.589057 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.589060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.589881 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.589890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.590489 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.590493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.590573 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.591505 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.591517 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.591875 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.593434 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.593437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.593439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.593615 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.593890 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.593893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.593895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.594013 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.594429 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.594432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.594514 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.595194 (+ 680us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.595305 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.62@KRBTEST.COM for oryx/127.2.123.61@KRBTEST.COM
I20260928 02:41:51.607409 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.599364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.2.123.61:40139)
0928 02:41:51.599469 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.599473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.600123 (+ 650us) server_negotiation.cc:407] Connection header received
0928 02:41:51.600290 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.600293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.600338 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.600399 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.601011 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.601471 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.602359 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.602574 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.602655 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.604650 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.604653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.604656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.604679 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.605819 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.606282 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.606285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.606287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.606355 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.606593 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.606596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.606598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.606751 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.606830 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.607156 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.607258 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:41:51.607416 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.599318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.0.0.1:38040)
0928 02:41:51.600010 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.600048 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.600173 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.600445 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.600452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.600801 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.600898 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.600904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.601567 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.601570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.602258 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.602268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.602665 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.602669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.602777 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.603218 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.603228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.604530 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.605914 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.605923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.605928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.606195 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.606432 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.606435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.606437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.606519 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.606845 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.606852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.606958 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.607083 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.607264 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":77,"threads_started":1}
I20260928 02:41:51.608948 8655 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.123.61:40139: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6"
member_type: NON_VOTER
last_known_addr {
host: "127.2.123.61"
port: 40139
}
attrs {
promote: true
}
}
dest_uuid: "4080b99d797d41a09e92d1d747845838"
cas_config_opid_index: -1
I20260928 02:41:51.609570 8655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } }
I20260928 02:41:51.611356 9213 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 02f615b6adfd4a0eace185decbe5f9f6 (127.2.123.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } }
I20260928 02:41:51.612944 9213 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.123.61:40139
I20260928 02:41:51.613046 9161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:51.613129 9214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } }
I20260928 02:41:51.613216 9214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:51.614359 9161 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Refusing update from remote peer 4080b99d797d41a09e92d1d747845838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:41:51.614952 9214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:51.615110 9119 master.cc:537] Master@127.2.123.61:40139 shutting down...
W20260928 02:41:51.616693 8641 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:41:51.617756 8641 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 -> Peer 02f615b6adfd4a0eace185decbe5f9f6 (127.2.123.61:40139): Couldn't send request to peer 02f615b6adfd4a0eace185decbe5f9f6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:41:51.619778 9119 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:51.620090 9119 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:51.620208 9119 tablet_replica.cc:330] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: stopping tablet replica
WARNING: no policy specified for oryx/127.2.123.60@KRBTEST.COM; defaulting to no policy
I20260928 02:41:51.624794 9119 master.cc:552] Master@127.2.123.61:40139 shutdown complete.
I20260928 02:41:51.624855 9119 master_runner.cc:305] Clearing existing system tablet
I20260928 02:41:51.625211 9119 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:51.625265 9119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.60@KRBTEST.COM" created.
I20260928 02:41:51.626590 9119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:51.627504 9219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.627753 9119 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:51.627889 9119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:51.627928 9119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
uuid: "02f615b6adfd4a0eace185decbe5f9f6"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "db23702846b1ae7a364cfb67fba999cf"
server_key_iv: "3cf69a408020f87cae782d724c7904a9"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.629769 9119 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:51.632215 9119 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:41:51.632316 9119 log.cc:1196] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:41:51.632555 9119 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Deleting consensus metadata
I20260928 02:41:51.632720 9119 master_runner.cc:315] Copying system tablet from 127.2.123.62:42551
I20260928 02:41:51.633924 9119 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: tablet copy: Beginning tablet copy session from remote peer at address 127.2.123.62:42551
I20260928 02:41:51.642678 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.634480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:59878)
0928 02:41:51.634894 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.634908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.635035 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.635320 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.635323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.635508 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.635624 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.635630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.636709 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.636711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.637453 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.637464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.638063 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.638066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.638160 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.638682 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.638694 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.639053 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.640900 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.640903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.640904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.641136 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.641448 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.641452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.641454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.641550 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.642077 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.642080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.642189 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.642295 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.642447 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":83,"threads_started":1}
I20260928 02:41:51.642689 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.634605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.2.123.62:42551)
0928 02:41:51.634746 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.634751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.635008 (+ 257us) server_negotiation.cc:407] Connection header received
0928 02:41:51.635126 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.635129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.635184 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.635268 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.635731 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.636575 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.637572 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.637933 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.638057 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.639182 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.639186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.639189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.639215 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.640769 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.641247 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.641251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.641253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.641356 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.641666 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.641671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.641673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.641904 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.642093 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.642368 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.642485 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:41:51.643227 8675 tablet_copy_service.cc:140] P 4080b99d797d41a09e92d1d747845838: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 02f615b6adfd4a0eace185decbe5f9f6 ({username='oryx', principal='oryx/127.2.123.61@KRBTEST.COM'} at 127.0.0.1:59878)
I20260928 02:41:51.643391 8675 tablet_copy_service.cc:161] P 4080b99d797d41a09e92d1d747845838: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 02f615b6adfd4a0eace185decbe5f9f6 at {username='oryx', principal='oryx/127.2.123.61@KRBTEST.COM'} at 127.0.0.1:59878: session id = 02f615b6adfd4a0eace185decbe5f9f6-00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.60.keytab.
Entry for principal oryx/127.2.123.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.60.keytab.
I20260928 02:41:51.645165 8675 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:41:51.645203 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.60:42831
--webserver_interface=127.2.123.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.60.keytab
--principal=oryx/127.2.123.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:42551,127.2.123.61:40139,127.2.123.60:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:51.646893 9119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:41:51.649348 9119 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: tablet copy: Starting download of 0 data blocks...
I20260928 02:41:51.649554 9119 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: tablet copy: Starting download of 1 WAL segments...
I20260928 02:41:51.651473 9119 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:41:51.653378 8675 tablet_copy_service.cc:342] P 4080b99d797d41a09e92d1d747845838: Request end of tablet copy session 02f615b6adfd4a0eace185decbe5f9f6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.123.61@KRBTEST.COM'} at 127.0.0.1:59878
I20260928 02:41:51.653509 8675 tablet_copy_service.cc:433] P 4080b99d797d41a09e92d1d747845838: ending tablet copy session 02f615b6adfd4a0eace185decbe5f9f6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 02f615b6adfd4a0eace185decbe5f9f6
I20260928 02:41:51.660849 9119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:51.662489 9231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:51.662751 9228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:51.662822 9119 server_base.cc:1034] running on GCE node
W20260928 02:41:51.663028 9229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:51.663216 9119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:51.663473 9119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:51.664609 9119 hybrid_clock.cc:648] HybridClock initialized: now 1790563311664602 us; error 24 us; skew 500 ppm
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:51.665941 9119 init.cc:377] Logged in from keytab as oryx/127.2.123.61@KRBTEST.COM (short username oryx)
I20260928 02:41:51.666359 9119 webserver.cc:460] Webserver started at http://127.2.123.61:46423/ using document root <none> and password file <none>
I20260928 02:41:51.666525 9119 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:51.666558 9119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:51.667526 9119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:51.668298 9238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.668497 9119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:51.668562 9119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
uuid: "02f615b6adfd4a0eace185decbe5f9f6"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "db23702846b1ae7a364cfb67fba999cf"
server_key_iv: "3cf69a408020f87cae782d724c7904a9"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.668637 9119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:51.682346 9119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:51.683977 9119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:51.688269 9119 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.61:40139
I20260928 02:41:51.688290 9290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.61:40139 every 8 connection(s)
I20260928 02:41:51.688771 9119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
I20260928 02:41:51.690594 9291 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:41:51.691360 9291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Bootstrap starting.
I20260928 02:41:51.698865 9291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:41:51.699137 9291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Bootstrap complete.
I20260928 02:41:51.699632 9291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } }
I20260928 02:41:51.699761 9291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 02f615b6adfd4a0eace185decbe5f9f6, State: Initialized, Role: LEARNER
I20260928 02:41:51.699858 9291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } }
I20260928 02:41:51.700196 9294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } }
I20260928 02:41:51.700297 9291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:51.700362 9294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:41:51.703413 9305 catalog_manager.cc:1259] Loaded cluster ID: 034e446c90f14499bdd89910a5cde6a9
I20260928 02:41:51.703464 9305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: loading cluster ID for follower catalog manager: success
I20260928 02:41:51.704993 9305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: acquiring CA information for follower catalog manager: success
I20260928 02:41:51.706074 9305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:41:51.792624 9227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:51.792912 9227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:51.792953 9227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:51.796947 9227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:51.797016 9227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:51.797034 9227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:51.797049 9227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:51.797063 9227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:51.802516 9227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:42551,127.2.123.61:40139,127.2.123.60:42831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.60.keytab
--principal=oryx/127.2.123.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.60:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.123.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:51.803637 9227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:51.804682 9227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:51.812845 9313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:51.812808 9311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:51.812815 9310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:51.813135 9227 server_base.cc:1034] running on GCE node
I20260928 02:41:51.813653 9227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:51.814749 9227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:51.815954 9227 hybrid_clock.cc:648] HybridClock initialized: now 1790563311815932 us; error 35 us; skew 500 ppm
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:51.818552 9227 init.cc:377] Logged in from keytab as oryx/127.2.123.60@KRBTEST.COM (short username oryx)
I20260928 02:41:51.819392 9227 webserver.cc:460] Webserver started at http://127.2.123.60:39441/ using document root <none> and password file <none>
I20260928 02:41:51.819929 9227 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:51.819989 9227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:51.820160 9227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:51.822147 9227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data/instance:
uuid: "fe641e889deb47638ae4e4c23f4dbb29"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "a8aab743aac3df1b6555f616859ee4b3"
server_key_iv: "b8fd6c3eaee80e71d0ad0dbffc01d319"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.822695 9227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal/instance:
uuid: "fe641e889deb47638ae4e4c23f4dbb29"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "a8aab743aac3df1b6555f616859ee4b3"
server_key_iv: "b8fd6c3eaee80e71d0ad0dbffc01d319"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.826370 9227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:51.828629 9320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.829937 9227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:51.830103 9227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
uuid: "fe641e889deb47638ae4e4c23f4dbb29"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "a8aab743aac3df1b6555f616859ee4b3"
server_key_iv: "b8fd6c3eaee80e71d0ad0dbffc01d319"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.830200 9227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:51.842303 9227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:51.845346 9227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:51.845515 9227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:51.852188 9227 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.60:42831
I20260928 02:41:51.852440 9372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.60:42831 every 8 connection(s)
I20260928 02:41:51.853153 9227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
I20260928 02:41:51.855893 9373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:51.859426 9373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:41:51.862787 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 9227
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.60@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.123.60@KRBTEST.COM
I20260928 02:41:51.873632 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.861301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.2.123.62:42551)
0928 02:41:51.861471 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.861475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.862210 (+ 735us) server_negotiation.cc:407] Connection header received
0928 02:41:51.863142 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.863145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.863202 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.863277 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.864544 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.865066 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.866343 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.866572 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.866670 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.869453 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.869457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.869459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.869488 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.870901 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.871622 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.871626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.871628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.871716 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.872126 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.872129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.872131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.872394 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.872503 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.873200 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.873387 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:41:51.874876 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.861583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:59890)
0928 02:41:51.862060 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.862111 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.862944 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.863493 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.863503 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.863934 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.864405 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.864418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.865189 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.865192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.866220 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.866229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.866710 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.866714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.866903 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.867441 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.867462 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.869293 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.871055 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.871064 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.871075 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.871495 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.871846 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.871849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.871851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.872022 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.872521 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.872528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.872879 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.873100 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.873456 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":178,"thread_start_us":104,"threads_started":1}
I20260928 02:41:51.876760 9373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:41:51.879444 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.863537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.2.123.60:42831)
0928 02:41:51.863855 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.863864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.863882 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:41:51.863914 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.863917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.864069 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.864204 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.864891 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.865748 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.866567 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.867003 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.867094 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.869573 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.869580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.869594 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.869615 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.871597 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.872091 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.872097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.872099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.877428 (+ 5329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.877798 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.877802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.877804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.878136 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.878982 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.879084 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.879246 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":238,"thread_start_us":151,"threads_started":1}
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.60@KRBTEST.COM for oryx/127.2.123.61@KRBTEST.COM
I20260928 02:41:51.887472 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.878227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.0.0.1:38044)
0928 02:41:51.878401 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.878415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.878482 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.879432 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.879435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.879816 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.879923 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.879929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.881167 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.881171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.881824 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.881833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.882380 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.882383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.882877 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.883275 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.883285 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.884545 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.886236 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.886239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.886241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.886430 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.886671 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.886673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.886674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.886741 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.887077 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.887080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.887161 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.887244 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.887348 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 02:41:51.887514 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.878350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.2.123.61:40139)
0928 02:41:51.879128 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.879132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.879144 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:51.879181 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.879184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.879232 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.879316 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.880048 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.881014 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.881947 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.882235 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.882354 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.884679 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.884683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.884685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.884710 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.886124 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.886527 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.886530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.886531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.886606 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.886810 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.886813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.886815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.886990 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.887065 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.887293 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.887390 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":153,"threads_started":1}
I20260928 02:41:51.888195 9373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563311, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.60@KRBTEST.COM for oryx/127.2.123.60@KRBTEST.COM
I20260928 02:41:51.895514 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.888638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.0.0.1:49686)
0928 02:41:51.888756 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.888771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.888843 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.889091 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.889093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.889219 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.889308 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.889319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.890384 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.890387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.890876 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.890883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.891301 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.891303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.891358 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.891719 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.891729 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.892874 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.894359 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.894362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.894363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.894548 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.894782 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.894784 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.894786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.894859 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.895147 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.895149 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.895214 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.895285 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.895382 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:41:51.895550 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.888813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.2.123.60:42831)
0928 02:41:51.888929 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.888932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.888943 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:41:51.888970 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.888972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.889003 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.889061 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.889404 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.889899 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.890971 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.891203 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.891279 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.892984 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.892988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.892989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.893009 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.894251 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.894620 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.894623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.894624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.894684 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.894930 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.894932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.894933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.895084 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.895151 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.895327 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.895433 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":51}
I20260928 02:41:51.897778 8846 heartbeater.cc:503] Master 127.2.123.62:42551 was elected leader, sending a full tablet report...
I20260928 02:41:51.898464 9373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Bootstrap starting.
I20260928 02:41:51.900974 9373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:51.901708 9373 log.cc:824] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:51.903636 9373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: No bootstrap required, opened a new log
I20260928 02:41:51.906599 9373 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } }
I20260928 02:41:51.906806 9373 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:51.906868 9373 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe641e889deb47638ae4e4c23f4dbb29, State: Initialized, Role: FOLLOWER
I20260928 02:41:51.907269 9373 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } }
I20260928 02:41:51.907667 9379 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } } }
I20260928 02:41:51.907778 9379 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:51.908360 9373 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:41:51.912796 9390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:41:51.912864 9390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:41:51.915884 9227 master_runner.cc:428] Detected that this master fe641e889deb47638ae4e4c23f4dbb29 is joining an existing cluster
I20260928 02:41:51.915948 9227 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.123.60:42831
I20260928 02:41:51.925992 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.918580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.2.123.60:42831)
0928 02:41:51.918669 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.918673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.918912 (+ 239us) server_negotiation.cc:407] Connection header received
0928 02:41:51.919062 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.919065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.919101 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.919169 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.919670 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.920421 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.921442 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.921768 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.921828 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.923412 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.923415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.923418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.923439 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.924622 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.924979 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.924983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.924985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.925046 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.925270 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.925273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.925275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.925421 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.925499 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.925706 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.925838 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":29}
I20260928 02:41:51.926005 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.918249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.0.0.1:49688)
0928 02:41:51.918790 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.918802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.918967 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.919195 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.919197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.919452 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.919574 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.919581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.920518 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.920520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.921329 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.921348 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.922001 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.922005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.922138 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.922876 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.922888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.923297 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.924713 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.924716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.924718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.924903 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.925117 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.925120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.925122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.925198 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.925499 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.925501 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.925578 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.925658 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.925787 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":98,"threads_started":1}
I20260928 02:41:51.932386 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 8717
I20260928 02:41:51.939198 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:39239
--local_ip_for_outbound_sockets=127.2.123.1
--tserver_master_addrs=127.2.123.62:42551
--webserver_port=37047
--webserver_interface=127.2.123.1
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab
--principal=oryx/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139,127.2.123.60:42831 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:51.946960 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.918638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.0.0.1:38048)
0928 02:41:51.918865 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.918877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.918954 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.919315 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.919318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.919571 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.919729 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.919737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.920902 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.920905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.922816 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.922826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.923534 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.923537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.923619 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.942510 (+ 18891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.942529 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.943077 (+ 548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.945135 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.945138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.945141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.945413 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.945703 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.945706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.945708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.945909 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.946425 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.946427 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.946520 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.946603 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.946750 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":21,"thread_start_us":69,"threads_started":1}
I20260928 02:41:51.947070 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.918921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.2.123.62:42551)
0928 02:41:51.919458 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.919462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.919475 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:41:51.919508 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.919511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.919552 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.919621 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.920142 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.920880 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.921933 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.922260 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.922331 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.943333 (+ 21002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.943338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.943340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.943373 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.945125 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.945641 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.945645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.945647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.945738 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.946083 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.946087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.946089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.946330 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.946404 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.946754 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.946906 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
I20260928 02:41:51.947598 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.918211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:59902)
0928 02:41:51.918517 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.918531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.918622 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.919720 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.919723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.919911 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.920015 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.920021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.921002 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.921005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.921812 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.921819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.922455 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.922458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.922518 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.942577 (+ 20059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.942593 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.943133 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.945260 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.945263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.945265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.945518 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.945867 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.945870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.945872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.945979 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.946477 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.946480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.946574 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.947273 (+ 699us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.947421 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":146,"threads_started":1}
I20260928 02:41:51.947908 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.919036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.2.123.61:40139)
0928 02:41:51.919136 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.919140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.919152 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:51.919187 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.919189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.919231 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.919304 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.919865 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.920736 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.922949 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.923320 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.923391 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.943297 (+ 19906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.943301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.943304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.943335 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.945003 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.945523 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.945527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.945529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.945608 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.946055 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.946058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.946060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.946299 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.946384 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.947639 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.947761 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 02:41:51 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563310, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.62@KRBTEST.COM for oryx/127.2.123.60@KRBTEST.COM
I20260928 02:41:51.960505 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.951960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.2.123.60:42831)
0928 02:41:51.952073 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.952077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.952089 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:51.952164 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.952167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.952218 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.952296 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:51.952717 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.953503 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.954705 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.955027 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.955124 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.957332 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:51.957336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:51.957339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:51.957363 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:51.958887 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.959345 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.959349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.959350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.959427 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:51.959684 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:51.959687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:51.959689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:51.959883 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:51.959969 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.960250 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.960357 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:41:51.960505 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.951602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.0.0.1:49692)
0928 02:41:51.951730 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.951745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.951829 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.952326 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.952330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.952512 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:51.952612 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.952618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.953660 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.953664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.954593 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.954604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.955138 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.955142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.955238 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.955669 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:51.955682 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:51.957202 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:51.958995 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.958998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.959000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.959253 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.959502 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:51.959505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:51.959507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:51.959594 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:51.959966 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:51.959969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:51.960068 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.960177 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.960330 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:41:51.961769 8655 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.123.60:42831: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29"
member_type: NON_VOTER
last_known_addr {
host: "127.2.123.60"
port: 42831
}
attrs {
promote: true
}
}
dest_uuid: "4080b99d797d41a09e92d1d747845838"
cas_config_opid_index: 5
I20260928 02:41:51.962572 8655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } }
I20260928 02:41:51.963636 9214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.965044 9214 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.123.60:42831
I20260928 02:41:51.965236 9214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.965335 9214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:51.965850 9227 master.cc:537] Master@127.2.123.60:42831 shutting down...
I20260928 02:41:51.966199 9347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:51.967657 9347 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 FOLLOWER]: Refusing update from remote peer 4080b99d797d41a09e92d1d747845838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:41:51.968837 9213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:41:51.971536 8640 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 -> Peer fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60:42831): Couldn't send request to peer fe641e889deb47638ae4e4c23f4dbb29. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:41:51.972285 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.963740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.0.0.1:38060)
0928 02:41:51.964357 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.964370 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:51.964433 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:51.967908 (+ 3475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:51.967912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:51.967931 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:51.968021 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.968027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.969256 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.969260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.970551 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:51.970562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.971863 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.971866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:51.971963 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.972009 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:41:51.972090 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:41:51.972161 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546}
I20260928 02:41:51.972910 9227 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:51.973068 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.963985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.2.123.61:40139)
0928 02:41:51.967630 (+ 3645us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:51.967634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:51.967650 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:41:51.967701 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:51.967703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:51.967763 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:51.967857 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:51.968224 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.968987 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.970687 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:51.971372 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:51.971467 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:51.971708 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:51.972531 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 02:41:51.972902 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3554}
I20260928 02:41:51.973181 9227 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:51.973279 9227 tablet_replica.cc:330] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: stopping tablet replica
I20260928 02:41:51.973824 9257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 LEARNER]: Refusing update from remote peer 4080b99d797d41a09e92d1d747845838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:41:51.974443 9213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:41:51.976230 9257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.978065 9401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.978258 9401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:41:51.978540 9294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4080b99d797d41a09e92d1d747845838. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: NON_VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: true } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.978657 9294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:41:51.983323 9213 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: attempting to promote NON_VOTER 02f615b6adfd4a0eace185decbe5f9f6 to VOTER
I20260928 02:41:51.983798 9213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } }
I20260928 02:41:51.985057 9257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 LEARNER]: Refusing update from remote peer 4080b99d797d41a09e92d1d747845838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:41:51.985441 9214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 02:41:51.985599 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.984801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.0.0.1:49704)
0928 02:41:51.985337 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.985443 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429}
I20260928 02:41:51.986495 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:51.986116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.0.0.1:49720)
0928 02:41:51.986377 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:51.986423 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
W20260928 02:41:51.986666 8640 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 -> Peer fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60:42831): Couldn't send request to peer fe641e889deb47638ae4e4c23f4dbb29. Status: Network error: Client connection negotiation failed: client connection to 127.2.123.60:42831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:41:51.987495 9214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 02f615b6adfd4a0eace185decbe5f9f6 (127.2.123.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.988968 9214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.989070 9214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:51.989190 9214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.989274 9214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:51.989856 9257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 02f615b6adfd4a0eace185decbe5f9f6 (127.2.123.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.991415 9227 master.cc:552] Master@127.2.123.60:42831 shutdown complete.
I20260928 02:41:51.991358 9294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:51.991463 9227 master_runner.cc:305] Clearing existing system tablet
I20260928 02:41:51.991462 9294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:51.991688 9227 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:51.991729 9227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:51.992983 9227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:51.993873 9405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:51.994117 9227 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:51.994268 9227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:51.994302 9227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
uuid: "fe641e889deb47638ae4e4c23f4dbb29"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "a8aab743aac3df1b6555f616859ee4b3"
server_key_iv: "b8fd6c3eaee80e71d0ad0dbffc01d319"
server_key_version: "encryptionkey@0"
I20260928 02:41:51.996285 9227 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:51.998689 9227 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:41:51.998777 9227 log.cc:1196] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:41:51.999001 9227 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Deleting consensus metadata
I20260928 02:41:51.999145 9227 master_runner.cc:315] Copying system tablet from 127.2.123.62:42551
I20260928 02:41:52.000255 9227 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: tablet copy: Beginning tablet copy session from remote peer at address 127.2.123.62:42551
I20260928 02:41:52.009171 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.000894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.2.123.62:42551)
0928 02:41:52.001018 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.001022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.001303 (+ 281us) server_negotiation.cc:407] Connection header received
0928 02:41:52.001415 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.001418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.001473 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.001560 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.002062 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.003002 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.004151 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.004474 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.004601 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.005813 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.005817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.005820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.005846 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.007423 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.007865 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.007868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.007871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.007957 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.008265 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.008268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.008270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.008482 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.008574 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.008857 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.009007 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:41:52.009790 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.000798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:59918)
0928 02:41:52.001198 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.001213 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.001331 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.001584 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.001587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.001801 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.001933 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.001939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.003113 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.003117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.004022 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.004032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.004604 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.004608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.004702 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.005244 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.005258 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.005668 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.007529 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.007532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.007534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.007774 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.008038 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.008041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.008043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.008129 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.008568 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.008571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.008668 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.008769 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.009482 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":87,"threads_started":1}
I20260928 02:41:52.010332 8675 tablet_copy_service.cc:140] P 4080b99d797d41a09e92d1d747845838: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fe641e889deb47638ae4e4c23f4dbb29 ({username='oryx', principal='oryx/127.2.123.60@KRBTEST.COM'} at 127.0.0.1:59918)
I20260928 02:41:52.010437 8675 tablet_copy_service.cc:161] P 4080b99d797d41a09e92d1d747845838: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fe641e889deb47638ae4e4c23f4dbb29 at {username='oryx', principal='oryx/127.2.123.60@KRBTEST.COM'} at 127.0.0.1:59918: session id = fe641e889deb47638ae4e4c23f4dbb29-00000000000000000000000000000000
I20260928 02:41:52.011536 8675 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:41:52.012784 9227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:41:52.015138 9227 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: tablet copy: Starting download of 0 data blocks...
I20260928 02:41:52.015328 9227 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: tablet copy: Starting download of 1 WAL segments...
I20260928 02:41:52.016741 9227 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:41:52.018555 8675 tablet_copy_service.cc:342] P 4080b99d797d41a09e92d1d747845838: Request end of tablet copy session fe641e889deb47638ae4e4c23f4dbb29-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.123.60@KRBTEST.COM'} at 127.0.0.1:59918
I20260928 02:41:52.018644 8675 tablet_copy_service.cc:433] P 4080b99d797d41a09e92d1d747845838: ending tablet copy session fe641e889deb47638ae4e4c23f4dbb29-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fe641e889deb47638ae4e4c23f4dbb29
I20260928 02:41:52.026404 9227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:52.028093 9412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:52.028345 9227 server_base.cc:1034] running on GCE node
W20260928 02:41:52.028371 9415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:52.028503 9413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:52.028789 9227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:52.029088 9227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:52.030229 9227 hybrid_clock.cc:648] HybridClock initialized: now 1790563312030213 us; error 41 us; skew 500 ppm
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:52.031769 9227 init.cc:377] Logged in from keytab as oryx/127.2.123.60@KRBTEST.COM (short username oryx)
I20260928 02:41:52.032227 9227 webserver.cc:460] Webserver started at http://127.2.123.60:38865/ using document root <none> and password file <none>
I20260928 02:41:52.032402 9227 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:52.032440 9227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:52.033919 9227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:52.034950 9422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.035386 9227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:52.035459 9227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
uuid: "fe641e889deb47638ae4e4c23f4dbb29"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "a8aab743aac3df1b6555f616859ee4b3"
server_key_iv: "b8fd6c3eaee80e71d0ad0dbffc01d319"
server_key_version: "encryptionkey@0"
I20260928 02:41:52.035543 9227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:52.062304 9227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:52.064594 9227 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260928 02:41:52.069279 9399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:52.069594 9399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:52.069657 9399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:41:52.070068 9227 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.60:42831
I20260928 02:41:52.070719 9227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
I20260928 02:41:52.070847 9477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.60:42831 every 8 connection(s)
I20260928 02:41:52.072202 9478 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:41:52.072996 9478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Bootstrap starting.
W20260928 02:41:52.075711 9399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:52.075773 9399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:52.075856 9399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:52.081926 9478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:41:52.082180 9478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: Bootstrap complete.
I20260928 02:41:52.082635 9478 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } }
I20260928 02:41:52.082789 9478 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fe641e889deb47638ae4e4c23f4dbb29, State: Initialized, Role: LEARNER
I20260928 02:41:52.082887 9478 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } }
I20260928 02:41:52.083175 9481 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:52.083303 9478 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:52.083310 9481 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:41:52.083561 9399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab
--principal=oryx/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:39239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=37047
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139,127.2.123.60:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:52.084892 9399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:52.085945 9492 catalog_manager.cc:1259] Loaded cluster ID: 034e446c90f14499bdd89910a5cde6a9
I20260928 02:41:52.086014 9492 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: loading cluster ID for follower catalog manager: success
I20260928 02:41:52.086441 9399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:52.087800 9492 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: acquiring CA information for follower catalog manager: success
I20260928 02:41:52.089066 9492 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:41:52.097769 9495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:52.097776 9494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:52.097790 9497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:52.097886 9399 server_base.cc:1034] running on GCE node
I20260928 02:41:52.098544 9399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:52.099220 9399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:52.100373 9399 hybrid_clock.cc:648] HybridClock initialized: now 1790563312100350 us; error 37 us; skew 500 ppm
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:52.103299 9399 init.cc:377] Logged in from keytab as oryx/127.2.123.1@KRBTEST.COM (short username oryx)
I20260928 02:41:52.104264 9399 webserver.cc:460] Webserver started at http://127.2.123.1:37047/ using document root <none> and password file <none>
I20260928 02:41:52.104817 9399 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:52.104897 9399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:52.108958 9399 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:52.111227 9504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.112493 8980 heartbeater.cc:503] Master 127.2.123.62:42551 was elected leader, sending a full tablet report...
I20260928 02:41:52.112504 9399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:52.112669 9399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "1ac7e976b8d44a7698eeb189152ba82f"
format_stamp: "Formatted at 2026-09-28 02:41:50 on dist-test-slave-rqvb"
server_key: "39245f4eea1b928a1b05a9d9bc7bd874"
server_key_iv: "2fb4c1a212836cdaf2b1de3b2918e5bf"
server_key_version: "encryptionkey@0"
I20260928 02:41:52.113341 9399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:52.130985 9399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:52.134393 9399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:52.134596 9399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:52.135218 9399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:52.136176 9399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:52.136227 9399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.136298 9399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:52.136332 9399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.146206 9399 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:39239
I20260928 02:41:52.146277 9617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:39239 every 8 connection(s)
I20260928 02:41:52.147284 9399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:52.157200 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 9399
I20260928 02:41:52.157394 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 8851
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.60@KRBTEST.COM
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.61@KRBTEST.COM
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:52.163916 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:41349
--local_ip_for_outbound_sockets=127.2.123.2
--tserver_master_addrs=127.2.123.62:42551
--webserver_port=33547
--webserver_interface=127.2.123.2
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab
--principal=oryx/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139,127.2.123.60:42831 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:41:52.164724 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.149222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:60715 (local address 127.2.123.62:42551)
0928 02:41:52.149368 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.149372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.150502 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 02:41:52.151259 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.151263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.151334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.151413 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.153754 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.154381 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.155467 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.155816 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.155929 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.160658 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.160662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.160665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.160697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.162616 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.163050 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.163054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.163056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.163139 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.163444 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.163448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.163449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.163680 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.163753 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.164419 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.164557 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:41:52.164726 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.149204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:46959 (local address 127.2.123.60:42831)
0928 02:41:52.149463 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.149475 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.150205 (+ 730us) server_negotiation.cc:407] Connection header received
0928 02:41:52.151262 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.151265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.151322 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.151378 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.152758 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.153798 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.155512 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.155880 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.155976 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.159475 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.159479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.159482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.159513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.161442 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.162111 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.162115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.162117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.162210 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.162620 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.162624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.162625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.162866 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.162947 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.164458 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.164575 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260928 02:41:52.165575 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.150910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:41825 (local address 127.2.123.61:40139)
0928 02:41:52.151032 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.151036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.151054 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:41:52.151261 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.151265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.151321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.151374 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.152735 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.153598 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.155503 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.155736 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.155795 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.161251 (+ 5456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.161256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.161259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.161290 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.162832 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.163373 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.163377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.163379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.163494 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.163933 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.163937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.163939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.164486 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.164558 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.165330 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.165438 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:41:52.166379 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.149550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.2.123.1:46959)
0928 02:41:52.150047 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.150100 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.151025 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.151604 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.151611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.152099 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.152591 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.152605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.153915 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.153918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.155358 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.155369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.156122 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.156126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.156242 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.156926 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.156950 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.159280 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.161591 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.161598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.161616 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.161944 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.162320 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.162324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.162326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.162511 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.163014 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.163021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.164274 (+ 1253us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.164915 (+ 641us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.165816 (+ 901us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":696,"spinlock_wait_cycles":47232,"thread_start_us":116,"threads_started":1}
I20260928 02:41:52.166363 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.150563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.2.123.1:41825)
0928 02:41:52.150816 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.150830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.151020 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.151604 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.151611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.152099 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.152591 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.152605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.153755 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.153758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.155358 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.155369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.155975 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.155979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.156291 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.156992 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.157006 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.161081 (+ 4075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.162953 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.162956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.162958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.163225 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.163667 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.163669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.163671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.163783 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.164636 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.164639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.164734 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.165274 (+ 540us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.165424 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":280,"spinlock_wait_cycles":87296,"thread_start_us":73,"threads_started":1}
I20260928 02:41:52.166682 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.149550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.1:60715)
0928 02:41:52.150390 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.150405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.151028 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.151602 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.151609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.152060 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.152589 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.152603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.154491 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.154495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.155339 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.155348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.155975 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.155979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.156286 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.157078 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.157091 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.160492 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.162723 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.162726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.162728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.162947 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.163251 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.163254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.163256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.163333 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.163825 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.163828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.164130 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.164885 (+ 755us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.165308 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":311,"spinlock_wait_cycles":13952,"thread_start_us":595,"threads_started":1}
I20260928 02:41:52.167740 9618 heartbeater.cc:347] Connected to a master server at 127.2.123.60:42831
I20260928 02:41:52.167804 9620 heartbeater.cc:347] Connected to a master server at 127.2.123.61:40139
I20260928 02:41:52.167804 9619 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42551
I20260928 02:41:52.168000 9618 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.167999 9620 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.167999 9619 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.168632 9619 heartbeater.cc:511] Master 127.2.123.62:42551 requested a full tablet report, sending...
I20260928 02:41:52.168632 9618 heartbeater.cc:511] Master 127.2.123.60:42831 requested a full tablet report, sending...
I20260928 02:41:52.168632 9620 heartbeater.cc:511] Master 127.2.123.61:40139 requested a full tablet report, sending...
I20260928 02:41:52.169894 8655 ts_manager.cc:194] Re-registered known tserver with Master: 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239)
I20260928 02:41:52.170118 9247 ts_manager.cc:194] Registered new tserver with Master: 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239)
I20260928 02:41:52.170207 9441 ts_manager.cc:194] Registered new tserver with Master: 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239)
I20260928 02:41:52.170681 8655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:60715
W20260928 02:41:52.277489 9626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:52.277760 9626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:52.277822 9626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:52.281749 9626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:52.281808 9626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:52.281888 9626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:52.287161 9626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab
--principal=oryx/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:41349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=33547
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139,127.2.123.60:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:52.288163 9626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:52.289119 9626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:52.298573 9631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:52.298619 9634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:52.298722 9626 server_base.cc:1034] running on GCE node
W20260928 02:41:52.298636 9632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:52.299199 9626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:52.299831 9626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:52.301008 9626 hybrid_clock.cc:648] HybridClock initialized: now 1790563312300982 us; error 46 us; skew 500 ppm
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:52.303792 9626 init.cc:377] Logged in from keytab as oryx/127.2.123.2@KRBTEST.COM (short username oryx)
I20260928 02:41:52.304648 9626 webserver.cc:460] Webserver started at http://127.2.123.2:33547/ using document root <none> and password file <none>
I20260928 02:41:52.305145 9626 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:52.305213 9626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:52.309013 9626 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:52.311075 9641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.312338 9626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:52.312440 9626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "cdb2f2879ec5457bab7a163cc9cb0059"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "0163c20e640c90ffdb10e08f14f9746f"
server_key_iv: "f8d6bce67447100d73f01da4c5ab13e7"
server_key_version: "encryptionkey@0"
I20260928 02:41:52.313019 9626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:52.326061 9626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:52.328411 9114 heartbeater.cc:503] Master 127.2.123.62:42551 was elected leader, sending a full tablet report...
I20260928 02:41:52.329231 9626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:52.329401 9626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:52.330132 9626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:52.330963 9626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:52.331012 9626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.331061 9626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:52.331090 9626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.339674 9626 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:41349
I20260928 02:41:52.339783 9754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:41349 every 8 connection(s)
I20260928 02:41:52.340667 9626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:52.342300 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 9626
I20260928 02:41:52.342492 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 8985
I20260928 02:41:52.350111 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:35061
--local_ip_for_outbound_sockets=127.2.123.3
--tserver_master_addrs=127.2.123.62:42551
--webserver_port=37539
--webserver_interface=127.2.123.3
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab
--principal=oryx/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139,127.2.123.60:42831 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.61@KRBTEST.COM
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.60@KRBTEST.COM
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:52.357654 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.344548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:46597 (local address 127.2.123.61:40139)
0928 02:41:52.344710 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.344715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.345020 (+ 305us) server_negotiation.cc:407] Connection header received
0928 02:41:52.345298 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.345301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.345355 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.345417 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.347351 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.347927 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.349576 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.349862 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.349992 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.353363 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.353368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.353371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.353410 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.355096 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.355754 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.355758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.355760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.355853 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.356304 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.356307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.356309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.356544 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.356621 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.357330 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.357469 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:41:52.358933 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.344657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.2.123.2:46597)
0928 02:41:52.344924 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.344939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.345085 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.345838 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.345847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.346499 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.347156 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.347177 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.348052 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.348055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.349452 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.349463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.350050 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.350055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.350509 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.351201 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.351223 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.353152 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.355214 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.355221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.355238 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.355645 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.355952 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.355955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.355956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.356167 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.356712 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.356718 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.357051 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.357275 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.357619 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"spinlock_wait_cycles":15232,"thread_start_us":106,"threads_started":1}
I20260928 02:41:52.359918 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.342509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:49623 (local address 127.2.123.60:42831)
0928 02:41:52.343099 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.343104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.344240 (+ 1136us) server_negotiation.cc:407] Connection header received
0928 02:41:52.345299 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.345302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.345355 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.345410 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.350087 (+ 4677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.350817 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.351986 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.352334 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.352421 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.356152 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.356156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.356158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.356204 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.358058 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.358541 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.358545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.358547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.358637 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.358956 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.358960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.358962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.359204 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.359307 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.359610 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.359756 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489}
I20260928 02:41:52.360519 9757 heartbeater.cc:347] Connected to a master server at 127.2.123.61:40139
I20260928 02:41:52.360634 9759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.342995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.2.123.2:49623)
0928 02:41:52.344067 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.344112 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.345064 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.345838 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.345847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.349786 (+ 3939us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.349896 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.349902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.350981 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.350985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.351817 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.351829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.352590 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.352594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.352730 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.353279 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.353296 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.355664 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.358154 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.358158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.358160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.358401 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.358761 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.358764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.358766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.358850 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.359299 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.359302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.359396 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.360249 (+ 853us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.360398 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":1,"thread_start_us":639,"threads_started":1}
I20260928 02:41:52.360778 9757 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.361364 9755 heartbeater.cc:347] Connected to a master server at 127.2.123.60:42831
I20260928 02:41:52.361373 9757 heartbeater.cc:511] Master 127.2.123.61:40139 requested a full tablet report, sending...
I20260928 02:41:52.361464 9755 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.361632 9755 heartbeater.cc:511] Master 127.2.123.60:42831 requested a full tablet report, sending...
I20260928 02:41:52.362090 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.345359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:33255 (local address 127.2.123.62:42551)
0928 02:41:52.345474 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.345478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.350340 (+ 4862us) server_negotiation.cc:407] Connection header received
0928 02:41:52.350491 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.350495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.350546 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.350596 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.351179 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.351898 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.353000 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.353320 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.353389 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.356136 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.356140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.356143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.356187 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.358833 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.359630 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.359634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.359636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.359730 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.360154 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.360158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.360160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.360386 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.360464 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.361757 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.361883 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:41:52.362545 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.345040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.2:33255)
0928 02:41:52.350256 (+ 5216us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.350271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.350399 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.350730 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.350733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.350903 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.351014 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.351020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.352052 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.352055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.352872 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.352882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.353532 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.353535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.353625 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.354089 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.354102 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.355998 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.358960 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.358963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.358965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.359520 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.359855 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.359858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.359860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.359961 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.361452 (+ 1491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.361455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.361552 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.362291 (+ 739us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.362428 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5140,"thread_start_us":77,"threads_started":1}
I20260928 02:41:52.362594 9441 ts_manager.cc:194] Registered new tserver with Master: cdb2f2879ec5457bab7a163cc9cb0059 (127.2.123.2:41349)
I20260928 02:41:52.363327 9247 ts_manager.cc:194] Registered new tserver with Master: cdb2f2879ec5457bab7a163cc9cb0059 (127.2.123.2:41349)
I20260928 02:41:52.363570 9756 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42551
I20260928 02:41:52.363654 9756 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.363834 9756 heartbeater.cc:511] Master 127.2.123.62:42551 requested a full tablet report, sending...
I20260928 02:41:52.364440 8655 ts_manager.cc:194] Re-registered known tserver with Master: cdb2f2879ec5457bab7a163cc9cb0059 (127.2.123.2:41349)
I20260928 02:41:52.365018 8655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:33255
W20260928 02:41:52.462669 9762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:52.462932 9762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:52.462988 9762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:52.466950 9762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:52.467005 9762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:52.467080 9762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:52.472321 9762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab
--principal=oryx/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:35061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=37539
--tserver_master_addrs=127.2.123.62:42551,127.2.123.61:40139,127.2.123.60:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:52.473425 9762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:52.474493 9762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:52.484246 9767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:52.484233 9768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:52.484221 9770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:52.484743 9762 server_base.cc:1034] running on GCE node
I20260928 02:41:52.485131 9762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:52.485790 9762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:52.486943 9762 hybrid_clock.cc:648] HybridClock initialized: now 1790563312486928 us; error 76 us; skew 500 ppm
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:52.489748 9762 init.cc:377] Logged in from keytab as oryx/127.2.123.3@KRBTEST.COM (short username oryx)
I20260928 02:41:52.490752 9762 webserver.cc:460] Webserver started at http://127.2.123.3:37539/ using document root <none> and password file <none>
I20260928 02:41:52.491429 9762 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:52.491537 9762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:52.494671 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.490936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49732 (local address 127.2.123.60:42831)
0928 02:41:52.491084 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.491089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.491142 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:41:52.491281 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.491284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.491348 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.491453 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:52.491740 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.492527 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.493531 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.494183 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.494297 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.494450 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.494490 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.494554 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:41:52.494691 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.490856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.0.0.1:49732)
0928 02:41:52.491078 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.491091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.491185 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.491458 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.491460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.491474 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:52.491584 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.491591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.492640 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.492643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.493431 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.493440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.494292 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.494295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.494365 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.494385 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.494470 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.494552 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":54,"threads_started":1}
I20260928 02:41:52.495502 9448 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 02:41:52.495867 9481 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4080b99d797d41a09e92d1d747845838. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:52.496042 9481 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:41:52.496346 9762 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 02:41:52.498343 9778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.499575 9762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:52.499706 9762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "79410553cd9a4050b052449aa189151e"
format_stamp: "Formatted at 2026-09-28 02:41:51 on dist-test-slave-rqvb"
server_key: "8c957e34c86a535b0cc9b09e67d02b8b"
server_key_iv: "76c87d1353cd2a316dcfe5617d5469e2"
server_key_version: "encryptionkey@0"
I20260928 02:41:52.500238 9762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:52.515944 9762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:52.519004 9762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:52.519168 9762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:52.519747 9762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:41:52.520577 9762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:52.520623 9762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.520663 9762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:52.520682 9762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:52.529217 9762 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:35061
I20260928 02:41:52.529245 9891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:35061 every 8 connection(s)
I20260928 02:41:52.530175 9762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:52.537168 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 9762
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.61@KRBTEST.COM
Sep 28 02:41:52 dist-test-slave-rqvb krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563312, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.60@KRBTEST.COM
I20260928 02:41:52.547621 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.531895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:48583 (local address 127.2.123.62:42551)
0928 02:41:52.532052 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.532056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.533206 (+ 1150us) server_negotiation.cc:407] Connection header received
0928 02:41:52.534265 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.534268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.534316 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.534374 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.536857 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.537416 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.538435 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.538672 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.538763 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.542441 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.542446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.542449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.542478 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.544831 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.545416 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.545420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.545422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.545523 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.546170 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.546173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.546175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.546383 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.546464 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.547338 (+ 874us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.547445 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:41:52.547626 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.533631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:54145 (local address 127.2.123.61:40139)
0928 02:41:52.533776 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.533780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.533926 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:41:52.534247 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.534252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.534308 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.534381 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.535867 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.536725 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.539129 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.539446 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.539523 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.543107 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.543110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.543113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.543137 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.544927 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.546204 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.546208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.546210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.546291 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.546614 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.546617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.546619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.546817 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.546897 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.547342 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.547453 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:41:52.548480 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.532423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.2.123.3:53941)
0928 02:41:52.533341 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.533353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.534027 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.534563 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.534572 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.535121 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.535742 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.535751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.536865 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.536868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.540168 (+ 3300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.540178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.541445 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.541448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.541513 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.541902 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.541933 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.543352 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.545729 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.545733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.545735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.546047 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.546337 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.546339 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.546341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.546451 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.546845 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.546848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.546932 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.547061 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.547431 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":135,"spinlock_wait_cycles":150016,"thread_start_us":165,"threads_started":1}
I20260928 02:41:52.549206 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.531895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:53941 (local address 127.2.123.60:42831)
0928 02:41:52.532021 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.532026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.533427 (+ 1401us) server_negotiation.cc:407] Connection header received
0928 02:41:52.534259 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.534262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.534318 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.534371 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.535900 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.536722 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.540890 (+ 4168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.541206 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.541292 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.543467 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.543471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.543473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.543499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.545639 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.546159 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.546162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.546164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.546245 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.546544 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.546547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.546549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.546729 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.546806 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.548938 (+ 2132us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.549063 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:41:52.550202 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.532423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.3:48583)
0928 02:41:52.532979 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.533021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.534027 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.534563 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.534572 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.535121 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.535746 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.535751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.537520 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.537523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.538323 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.538332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.538810 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.538814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.539008 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.539693 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.539714 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.542243 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.544950 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.544957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.544976 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.545311 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.545615 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.545618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.545620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.546075 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.546526 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.546533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.546862 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.547061 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.549747 (+ 2686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":2461,"spinlock_wait_cycles":7680,"thread_start_us":647,"threads_started":1}
I20260928 02:41:52.550531 9892 heartbeater.cc:347] Connected to a master server at 127.2.123.60:42831
I20260928 02:41:52.550770 9892 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.550938 9893 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42551
I20260928 02:41:52.551007 9893 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.551409 9892 heartbeater.cc:511] Master 127.2.123.60:42831 requested a full tablet report, sending...
I20260928 02:41:52.551659 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.533573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.2.123.3:54145)
0928 02:41:52.533848 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.533858 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.534030 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.534619 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.534622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.535118 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:52.535734 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.535748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.538041 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.538045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.538920 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.538931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.539680 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.539683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.539768 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.540322 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:52.540336 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:52.542964 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:52.545764 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.545767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.545769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.546091 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.546391 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:52.546394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:52.546396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:52.546502 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:52.546939 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:52.546942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:52.547030 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.547119 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.551446 (+ 4327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":4132,"spinlock_wait_cycles":18560,"thread_start_us":78,"threads_started":1}
I20260928 02:41:52.551982 9893 heartbeater.cc:511] Master 127.2.123.62:42551 requested a full tablet report, sending...
I20260928 02:41:52.552399 9894 heartbeater.cc:347] Connected to a master server at 127.2.123.61:40139
I20260928 02:41:52.552469 9894 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:52.552611 9894 heartbeater.cc:511] Master 127.2.123.61:40139 requested a full tablet report, sending...
I20260928 02:41:52.552724 8655 ts_manager.cc:194] Re-registered known tserver with Master: 79410553cd9a4050b052449aa189151e (127.2.123.3:35061)
I20260928 02:41:52.553380 8655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:48583
I20260928 02:41:52.553412 9441 ts_manager.cc:194] Registered new tserver with Master: 79410553cd9a4050b052449aa189151e (127.2.123.3:35061)
I20260928 02:41:52.553988 9247 ts_manager.cc:194] Registered new tserver with Master: 79410553cd9a4050b052449aa189151e (127.2.123.3:35061)
I20260928 02:41:52.555117 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.539956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.2.123.61:40139)
0928 02:41:52.541703 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.541769 (+ 66us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.541784 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:41:52.541820 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.541822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.541866 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.541944 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.542786 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.543565 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.544722 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.545083 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.545190 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.547021 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.547024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.547027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.547084 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.548856 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.549274 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.549277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.549279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.549361 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.549648 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.549651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.549653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.549846 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.549984 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.554841 (+ 4857us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.554962 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1677,"thread_start_us":94,"threads_started":1}
I20260928 02:41:52.556325 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.539241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.2.123.62:42551)
0928 02:41:52.539471 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.539475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.540169 (+ 694us) server_negotiation.cc:407] Connection header received
0928 02:41:52.540364 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.540367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.540420 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.540482 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.541602 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.542658 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.550638 (+ 7980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.550988 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.551074 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.552782 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.552786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.552788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.552812 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.554365 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.554791 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.554795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.554797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.554885 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.555514 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.555518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.555519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.555735 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.555861 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.556058 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.556179 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260928 02:41:52.561199 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.540578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.2.123.60:42831)
0928 02:41:52.541090 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.541101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.543668 (+ 2567us) server_negotiation.cc:407] Connection header received
0928 02:41:52.543710 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.543713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.543763 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.543841 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.544736 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.545460 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.551080 (+ 5620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.551401 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.551492 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.558473 (+ 6981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.558477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.558480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.558504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.559766 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.560136 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.560139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.560140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.560207 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.560438 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.560441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.560443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.560605 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.560716 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.560926 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.561034 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":74,"threads_started":1}
I20260928 02:41:52.558588 8655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:52.562342 8655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:52.564867 9402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: NON_VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: true } } }
I20260928 02:41:52.564996 9402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:52.568130 9910 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: attempting to promote NON_VOTER fe641e889deb47638ae4e4c23f4dbb29 to VOTER
I20260928 02:41:52.568420 9910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } }
I20260928 02:41:52.569430 9448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 LEARNER]: Refusing update from remote peer 4080b99d797d41a09e92d1d747845838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 02:41:52.569537 9257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Refusing update from remote peer 4080b99d797d41a09e92d1d747845838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 02:41:52.569746 9911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 02:41:52.569866 9402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 02:41:52.571328 9402 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:52.571928 9448 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:52.572780 9911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:52.572876 9911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:52.572816 9402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:52.572922 9402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:52.572991 9911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:52.573073 9911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:52.573494 9257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:52.574508 9481 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:52.574601 9481 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:52.574944 9920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4080b99d797d41a09e92d1d747845838" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:52.575066 9920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:52.583408 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.579227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.2.123.3:35061)
0928 02:41:52.579622 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.579633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.579654 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:41:52.579726 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.579732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.579889 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.580235 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:52.580357 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.581227 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.582245 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.582809 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.582904 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.583109 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.583195 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.583281 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":94,"threads_started":1}
I20260928 02:41:52.583397 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.578473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35061 (local address 127.0.0.1:44524)
0928 02:41:52.578938 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.578952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.579029 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.580038 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.580041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.580053 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:52.580115 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.580120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.581327 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.581330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.582142 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.582151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.582915 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.582918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.583077 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.583100 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.583169 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.583241 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":31}
I20260928 02:41:52.584793 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.578840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:39239 (local address 127.0.0.1:33828)
0928 02:41:52.579512 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.579524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.579623 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.580106 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.580109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.580128 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:52.580213 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.580219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.581575 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.581577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.583083 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.583091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.584303 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.584306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.584457 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.584516 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.584591 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.584661 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":210,"thread_start_us":104,"threads_started":1}
I20260928 02:41:52.584856 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.578825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.2.123.2:41349)
0928 02:41:52.579125 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.579138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.579343 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:41:52.579555 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.579561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.579782 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.579947 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:52.580260 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.581438 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.583143 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.583962 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.584058 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.584189 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.584645 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.584738 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
I20260928 02:41:52.584913 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.579175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.2.123.1:39239)
0928 02:41:52.579440 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.579451 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.579620 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:41:52.579719 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.579723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.579867 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.579980 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:52.580333 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.581438 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.583193 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.583962 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.584070 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.584189 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.584633 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.584742 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20260928 02:41:52.585173 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.578684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41349 (local address 127.0.0.1:33988)
0928 02:41:52.579254 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.579266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.579407 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.579917 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.579920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.579935 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:52.580046 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.580052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.581566 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.581569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.582997 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.583008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.584286 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.584289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.584461 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.584519 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.585013 (+ 494us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.585087 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":166,"thread_start_us":95,"threads_started":1}
I20260928 02:41:52.586738 9826 tablet_service.cc:1467] Processing CreateTablet for tablet e7388b1b23614303ad74133d59b398ab (DEFAULT_TABLE table=test-table [id=a687ba91500340aa9a7da4ce09f868bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:52.587842 9826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7388b1b23614303ad74133d59b398ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:52.587652 9689 tablet_service.cc:1467] Processing CreateTablet for tablet e7388b1b23614303ad74133d59b398ab (DEFAULT_TABLE table=test-table [id=a687ba91500340aa9a7da4ce09f868bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:52.588619 9689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7388b1b23614303ad74133d59b398ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:52.588409 9552 tablet_service.cc:1467] Processing CreateTablet for tablet e7388b1b23614303ad74133d59b398ab (DEFAULT_TABLE table=test-table [id=a687ba91500340aa9a7da4ce09f868bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:52.589308 9552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7388b1b23614303ad74133d59b398ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:52.595227 9926 tablet_bootstrap.cc:492] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: Bootstrap starting.
I20260928 02:41:52.596361 9927 tablet_bootstrap.cc:492] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Bootstrap starting.
I20260928 02:41:52.597643 9926 tablet_bootstrap.cc:654] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:52.598090 9927 tablet_bootstrap.cc:654] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:52.598296 9928 tablet_bootstrap.cc:492] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Bootstrap starting.
I20260928 02:41:52.598914 9926 log.cc:824] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:52.598928 9927 log.cc:824] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:52.600178 9928 tablet_bootstrap.cc:654] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:52.600916 9927 tablet_bootstrap.cc:492] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: No bootstrap required, opened a new log
I20260928 02:41:52.600955 9926 tablet_bootstrap.cc:492] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: No bootstrap required, opened a new log
I20260928 02:41:52.600960 9928 log.cc:824] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:52.601125 9926 ts_tablet_manager.cc:1397] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:41:52.601141 9927 ts_tablet_manager.cc:1397] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 02:41:52.602913 9928 tablet_bootstrap.cc:492] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: No bootstrap required, opened a new log
I20260928 02:41:52.603111 9928 ts_tablet_manager.cc:1397] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:41:52.605482 9927 raft_consensus.cc:348] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.605753 9927 raft_consensus.cc:374] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:52.605814 9927 raft_consensus.cc:729] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79410553cd9a4050b052449aa189151e, State: Initialized, Role: FOLLOWER
I20260928 02:41:52.605877 9926 raft_consensus.cc:348] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.606132 9926 raft_consensus.cc:374] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:52.606197 9926 raft_consensus.cc:729] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb2f2879ec5457bab7a163cc9cb0059, State: Initialized, Role: FOLLOWER
I20260928 02:41:52.606287 9927 consensus_queue.cc:260] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.606600 9926 consensus_queue.cc:260] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.607538 9926 ts_tablet_manager.cc:1428] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260928 02:41:52.607547 9928 raft_consensus.cc:348] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.607692 9893 heartbeater.cc:503] Master 127.2.123.62:42551 was elected leader, sending a full tablet report...
I20260928 02:41:52.607764 9928 raft_consensus.cc:374] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:52.607821 9928 raft_consensus.cc:729] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac7e976b8d44a7698eeb189152ba82f, State: Initialized, Role: FOLLOWER
I20260928 02:41:52.608233 9928 consensus_queue.cc:260] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.609136 9928 ts_tablet_manager.cc:1428] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:41:52.609746 9756 heartbeater.cc:503] Master 127.2.123.62:42551 was elected leader, sending a full tablet report...
I20260928 02:41:52.611222 9619 heartbeater.cc:503] Master 127.2.123.62:42551 was elected leader, sending a full tablet report...
I20260928 02:41:52.611382 9927 ts_tablet_manager.cc:1428] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260928 02:41:52.649304 9621 tablet.cc:2369] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:52.722414 9933 raft_consensus.cc:482] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:52.722656 9933 raft_consensus.cc:504] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.723767 9933 leader_election.cc:288] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79410553cd9a4050b052449aa189151e (127.2.123.3:35061), 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239)
I20260928 02:41:52.728691 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.724045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:45781 (local address 127.2.123.3:35061)
0928 02:41:52.724212 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.724217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.724234 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:41:52.724400 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.724403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.724455 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.724532 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:52.724915 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.725672 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.727378 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.728009 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.728120 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.728181 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.728480 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.728570 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:41:52.728713 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.724124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:40503 (local address 127.2.123.1:39239)
0928 02:41:52.724225 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.724228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.724251 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:41:52.724417 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.724420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.724464 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.724537 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:52.724901 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.725729 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.727435 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.728020 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.728120 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.728181 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.728514 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.728613 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:41:52.728684 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.724024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:39239 (local address 127.2.123.2:40503)
0928 02:41:52.724176 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.724191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.724288 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.724558 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.724560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.724585 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:52.724801 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.724806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.725816 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.725819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.727230 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.727241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.728133 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.728137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.728299 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.728342 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.728477 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.728531 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":157,"spinlock_wait_cycles":8832}
I20260928 02:41:52.728684 9759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.723992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35061 (local address 127.2.123.2:45781)
0928 02:41:52.724176 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:52.724191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:52.724288 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:52.724558 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:52.724560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:52.724585 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:52.724797 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.724804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.725797 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.725801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.727230 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:52.727241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.728133 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.728137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:52.728296 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.728334 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:41:52.728412 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:41:52.728494 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":205,"spinlock_wait_cycles":14208}
I20260928 02:41:52.729383 9572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7388b1b23614303ad74133d59b398ab" candidate_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac7e976b8d44a7698eeb189152ba82f" is_pre_election: true
I20260928 02:41:52.729393 9846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7388b1b23614303ad74133d59b398ab" candidate_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79410553cd9a4050b052449aa189151e" is_pre_election: true
I20260928 02:41:52.729705 9572 raft_consensus.cc:2454] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdb2f2879ec5457bab7a163cc9cb0059 in term 0.
I20260928 02:41:52.729705 9846 raft_consensus.cc:2454] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdb2f2879ec5457bab7a163cc9cb0059 in term 0.
I20260928 02:41:52.730245 9642 leader_election.cc:302] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79410553cd9a4050b052449aa189151e, cdb2f2879ec5457bab7a163cc9cb0059; no voters:
I20260928 02:41:52.730511 9933 raft_consensus.cc:2781] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:52.730592 9933 raft_consensus.cc:482] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:52.730640 9933 raft_consensus.cc:3037] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:52.731680 9933 raft_consensus.cc:504] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.732004 9933 leader_election.cc:288] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [CANDIDATE]: Term 1 election: Requested vote from peers 79410553cd9a4050b052449aa189151e (127.2.123.3:35061), 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239)
I20260928 02:41:52.732414 9846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7388b1b23614303ad74133d59b398ab" candidate_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79410553cd9a4050b052449aa189151e"
I20260928 02:41:52.732539 9846 raft_consensus.cc:3037] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:52.733553 9846 raft_consensus.cc:2454] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdb2f2879ec5457bab7a163cc9cb0059 in term 1.
I20260928 02:41:52.733765 9572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7388b1b23614303ad74133d59b398ab" candidate_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac7e976b8d44a7698eeb189152ba82f"
I20260928 02:41:52.733829 9642 leader_election.cc:302] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79410553cd9a4050b052449aa189151e, cdb2f2879ec5457bab7a163cc9cb0059; no voters:
I20260928 02:41:52.733887 9572 raft_consensus.cc:3037] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:52.734012 9933 raft_consensus.cc:2781] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:52.734225 9933 raft_consensus.cc:686] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 LEADER]: Becoming Leader. State: Replica: cdb2f2879ec5457bab7a163cc9cb0059, State: Running, Role: LEADER
I20260928 02:41:52.734637 9933 consensus_queue.cc:237] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:52.735110 9572 raft_consensus.cc:2454] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdb2f2879ec5457bab7a163cc9cb0059 in term 1.
I20260928 02:41:52.738188 8654 catalog_manager.cc:5495] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdb2f2879ec5457bab7a163cc9cb0059 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:52.771126 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.767704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.2.123.2:41349)
0928 02:41:52.767852 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.767856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.767871 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:41:52.767946 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.767948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.768005 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.768106 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:52.768339 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.768855 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.769708 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.769932 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.770037 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.770331 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:52.770747 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:52.770853 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.770922 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.770979 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:41:52.776891 9846 raft_consensus.cc:1261] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 1 FOLLOWER]: Refusing update from remote peer cdb2f2879ec5457bab7a163cc9cb0059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:52.776924 9572 raft_consensus.cc:1261] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 1 FOLLOWER]: Refusing update from remote peer cdb2f2879ec5457bab7a163cc9cb0059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:52.777606 9933 consensus_queue.cc:1035] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:52.777781 9935 consensus_queue.cc:1035] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:41:52.781816 9895 tablet.cc:2369] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:52.786972 9940 mvcc.cc:205] Tried to move back new op lower bound from 7334147329127895040 to 7334147328965222400. Current Snapshot: MvccSnapshot[applied={T|T < 7334147329127895040}]
I20260928 02:41:52.796277 8654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:52.796438 8654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:52.800480 8654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4080b99d797d41a09e92d1d747845838: Sending DeleteTablet for 3 replicas of tablet e7388b1b23614303ad74133d59b398ab
I20260928 02:41:52.801429 9689 tablet_service.cc:1514] Processing DeleteTablet for tablet e7388b1b23614303ad74133d59b398ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:52 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:33988
I20260928 02:41:52.801440 9826 tablet_service.cc:1514] Processing DeleteTablet for tablet e7388b1b23614303ad74133d59b398ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:52 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:44524
I20260928 02:41:52.802024 9948 tablet_replica.cc:330] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: stopping tablet replica
I20260928 02:41:52.802380 9948 raft_consensus.cc:2229] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:52.802429 9552 tablet_service.cc:1514] Processing DeleteTablet for tablet e7388b1b23614303ad74133d59b398ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:52 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:33828
I20260928 02:41:52.802776 9948 raft_consensus.cc:2258] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:52.803054 9949 tablet_replica.cc:330] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: stopping tablet replica
I20260928 02:41:52.803529 9949 raft_consensus.cc:2229] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:52.803835 9949 raft_consensus.cc:2258] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:52.804451 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 8623
I20260928 02:41:52.805476 9948 ts_tablet_manager.cc:1905] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:52.805558 9949 ts_tablet_manager.cc:1905] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:52.808054 9948 ts_tablet_manager.cc:1918] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:52.808084 9949 ts_tablet_manager.cc:1918] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:52.808145 9948 log.cc:1196] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/e7388b1b23614303ad74133d59b398ab
I20260928 02:41:52.808176 9949 log.cc:1196] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/e7388b1b23614303ad74133d59b398ab
I20260928 02:41:52.808439 9948 ts_tablet_manager.cc:1939] T e7388b1b23614303ad74133d59b398ab P 79410553cd9a4050b052449aa189151e: Deleting consensus metadata
I20260928 02:41:52.808467 9949 ts_tablet_manager.cc:1939] T e7388b1b23614303ad74133d59b398ab P 1ac7e976b8d44a7698eeb189152ba82f: Deleting consensus metadata
I20260928 02:41:52.810026 9947 tablet_replica.cc:330] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: stopping tablet replica
I20260928 02:41:52.810551 9947 raft_consensus.cc:2229] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:52.810971 9947 raft_consensus.cc:2258] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:41:52.812352 9505 connection.cc:535] server connection from 127.0.0.1:33828 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:41:52.812502 9947 ts_tablet_manager.cc:1905] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:41:52.813130 9781 connection.cc:535] server connection from 127.0.0.1:44524 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:41:52.815505 9947 ts_tablet_manager.cc:1918] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:52.815600 9947 log.cc:1196] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/e7388b1b23614303ad74133d59b398ab
I20260928 02:41:52.815918 9947 ts_tablet_manager.cc:1939] T e7388b1b23614303ad74133d59b398ab P cdb2f2879ec5457bab7a163cc9cb0059: Deleting consensus metadata
W20260928 02:41:52.817200 9642 connection.cc:441] server connection from 127.0.0.1:33988 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33988 (request call id 1) could send its response
I20260928 02:41:52.819691 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.803432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.2.123.61:40139)
0928 02:41:52.803556 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.803560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.809739 (+ 6179us) server_negotiation.cc:407] Connection header received
0928 02:41:52.809849 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.809852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.809904 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.810007 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.811157 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.811978 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.813291 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.813648 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.813725 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.816075 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.816079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.816082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.816121 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.817794 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.818308 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.818311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.818313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.818402 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.818701 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.818705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.818707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.818964 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.819076 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.819404 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.819528 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
I20260928 02:41:52.826932 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.817733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.2.123.60:42831)
0928 02:41:52.817843 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.817847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.818453 (+ 606us) server_negotiation.cc:407] Connection header received
0928 02:41:52.818870 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.818873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.818918 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.818969 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.819888 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.820601 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.821661 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.822017 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.822089 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.823765 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.823768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.823771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.823791 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.825277 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.825728 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.825732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.825733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.825811 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.826143 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.826146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.826148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.826309 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.826426 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.826634 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.826774 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:41:52.829447 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.817881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.2.123.61:40139)
0928 02:41:52.817996 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.818000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.818621 (+ 621us) server_negotiation.cc:407] Connection header received
0928 02:41:52.818738 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.818741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.818788 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.818852 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.819701 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.820481 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.822544 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.823148 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.823223 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.825109 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.825113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.825115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.825139 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.827728 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.828185 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.828187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.828188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.828259 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.828613 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.828617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.828619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.828818 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.828957 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.829133 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.829340 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:41:52.830466 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:52.803271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.2.123.60:42831)
0928 02:41:52.803379 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:52.803383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:52.820504 (+ 17121us) server_negotiation.cc:407] Connection header received
0928 02:41:52.820641 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:52.820644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:52.820690 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:52.820733 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:52.821583 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.822308 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.823320 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:52.823658 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:52.823724 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:52.825224 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:52.825228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:52.825231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:52.825256 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:52.827038 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.828450 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.828453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.828454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.828522 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:52.829190 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:52.829193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:52.829195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:52.829387 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:52.829476 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:52.830206 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 02:41:52.830323 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:41:53.789392 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:53.788714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.1:41079)
0928 02:41:53.789163 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:53.789268 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":81,"threads_started":1}
W20260928 02:41:53.790478 9619 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:42551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
I20260928 02:41:53.791270 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:53.790655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.3:41227)
0928 02:41:53.791075 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:53.791166 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":85,"threads_started":1}
W20260928 02:41:53.791571 9893 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:42551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
I20260928 02:41:53.791929 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:53.790840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.2:50765)
0928 02:41:53.791760 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:53.791847 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":95,"threads_started":1}
W20260928 02:41:53.792179 9756 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:42551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
I20260928 02:41:54.300385 9961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4080b99d797d41a09e92d1d747845838)
I20260928 02:41:54.300621 9961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } }
I20260928 02:41:54.301349 9962 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4080b99d797d41a09e92d1d747845838)
I20260928 02:41:54.301494 9962 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } }
I20260928 02:41:54.301726 9961 leader_election.cc:288] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4080b99d797d41a09e92d1d747845838 (127.2.123.62:42551), fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60:42831)
I20260928 02:41:54.302414 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.301890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60372)
0928 02:41:54.302235 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.302316 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
W20260928 02:41:54.302583 9241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
I20260928 02:41:54.303330 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.303137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60384)
0928 02:41:54.303226 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.303269 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:41:54.303447 9962 leader_election.cc:288] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4080b99d797d41a09e92d1d747845838 (127.2.123.62:42551), 02f615b6adfd4a0eace185decbe5f9f6 (127.2.123.61:40139)
W20260928 02:41:54.303463 9241 leader_election.cc:334] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4080b99d797d41a09e92d1d747845838 (127.2.123.62:42551): Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
I20260928 02:41:54.303656 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.302682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60374)
0928 02:41:54.303025 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.303131 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":56,"threads_started":1}
W20260928 02:41:54.303799 9425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
I20260928 02:41:54.304543 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.304338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60386)
0928 02:41:54.304424 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.304476 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 02:41:54.304637 9425 leader_election.cc:334] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4080b99d797d41a09e92d1d747845838 (127.2.123.62:42551): Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
I20260928 02:41:54.306360 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.301987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:42831 (local address 127.0.0.1:49760)
0928 02:41:54.302344 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.302357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:54.302477 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:54.302813 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:54.302816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:54.302835 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:54.302987 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.302996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.303996 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.304000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.305081 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.305090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.306073 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.306076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.306141 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.306161 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:41:54.306220 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:41:54.306273 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":67,"threads_started":1}
I20260928 02:41:54.306432 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.302133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.2.123.60:42831)
0928 02:41:54.302348 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:54.302353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:54.302431 (+ 78us) server_negotiation.cc:407] Connection header received
0928 02:41:54.302599 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:54.302601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:54.302640 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:54.302708 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:54.303125 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.303856 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.305162 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.305901 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.306057 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.306114 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:54.306245 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:41:54.306314 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":144,"thread_start_us":51,"threads_started":1}
I20260928 02:41:54.306847 9448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02f615b6adfd4a0eace185decbe5f9f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fe641e889deb47638ae4e4c23f4dbb29" is_pre_election: true
I20260928 02:41:54.307051 9448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f615b6adfd4a0eace185decbe5f9f6 in term 1.
I20260928 02:41:54.307417 9240 leader_election.cc:302] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02f615b6adfd4a0eace185decbe5f9f6, fe641e889deb47638ae4e4c23f4dbb29; no voters: 4080b99d797d41a09e92d1d747845838
I20260928 02:41:54.307636 9961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:41:54.307657 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.303287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.2.123.61:40139)
0928 02:41:54.303917 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:54.303927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:54.303939 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:54.303978 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:54.303981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:54.304097 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:54.304185 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:54.304473 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.305039 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.306593 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.307149 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.307266 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.307319 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:54.307479 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:41:54.307555 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":89,"threads_started":1}
I20260928 02:41:54.307710 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.303067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:40139 (local address 127.0.0.1:38092)
0928 02:41:54.303641 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.303654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:54.303736 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:54.304285 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:54.304288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:54.304304 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:54.304386 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.304393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.305135 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.305138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.306459 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.306469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.307278 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.307282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.307374 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.307397 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:41:54.307456 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:41:54.307520 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":56,"thread_start_us":46,"threads_started":1}
I20260928 02:41:54.307714 9961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4080b99d797d41a09e92d1d747845838)
I20260928 02:41:54.307758 9961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:41:54.308073 9257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe641e889deb47638ae4e4c23f4dbb29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02f615b6adfd4a0eace185decbe5f9f6" is_pre_election: true
I20260928 02:41:54.308737 9961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } }
I20260928 02:41:54.308965 9257 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe641e889deb47638ae4e4c23f4dbb29 in current term 2: Already voted for candidate 02f615b6adfd4a0eace185decbe5f9f6 in this term.
I20260928 02:41:54.309057 9961 leader_election.cc:288] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [CANDIDATE]: Term 2 election: Requested vote from peers 4080b99d797d41a09e92d1d747845838 (127.2.123.62:42551), fe641e889deb47638ae4e4c23f4dbb29 (127.2.123.60:42831)
I20260928 02:41:54.309405 9448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02f615b6adfd4a0eace185decbe5f9f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fe641e889deb47638ae4e4c23f4dbb29"
I20260928 02:41:54.309403 9425 leader_election.cc:302] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe641e889deb47638ae4e4c23f4dbb29; no voters: 02f615b6adfd4a0eace185decbe5f9f6, 4080b99d797d41a09e92d1d747845838
I20260928 02:41:54.309544 9448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:41:54.309707 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.309174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60402)
0928 02:41:54.309595 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.309649 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
I20260928 02:41:54.310256 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.310104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60404)
0928 02:41:54.310170 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.310210 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 02:41:54.310353 9241 leader_election.cc:334] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4080b99d797d41a09e92d1d747845838 (127.2.123.62:42551): Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
I20260928 02:41:54.310665 9448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f615b6adfd4a0eace185decbe5f9f6 in term 2.
I20260928 02:41:54.310766 9962 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:41:54.310956 9240 leader_election.cc:302] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02f615b6adfd4a0eace185decbe5f9f6, fe641e889deb47638ae4e4c23f4dbb29; no voters: 4080b99d797d41a09e92d1d747845838
I20260928 02:41:54.311123 9961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:41:54.311338 9961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [term 2 LEADER]: Becoming Leader. State: Replica: 02f615b6adfd4a0eace185decbe5f9f6, State: Running, Role: LEADER
I20260928 02:41:54.311558 9961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } }
I20260928 02:41:54.312492 9971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02f615b6adfd4a0eace185decbe5f9f6. Latest consensus state: current_term: 2 leader_uuid: "02f615b6adfd4a0eace185decbe5f9f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:54.312621 9971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:54.312906 9973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:54.315256 9973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:54.315680 9973 catalog_manager.cc:1259] Loaded cluster ID: 034e446c90f14499bdd89910a5cde6a9
I20260928 02:41:54.315730 9973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:54.316318 9973 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:54.316756 9973 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Loaded TSK: 0
I20260928 02:41:54.317503 9973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:41:54.340875 9247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:54.344383 9247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:54.347257 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.346955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60422)
0928 02:41:54.347099 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.347169 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:41:54.347334 9448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [term 2 FOLLOWER]: Refusing update from remote peer 02f615b6adfd4a0eace185decbe5f9f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 02:41:54.347949 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.347721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60436)
0928 02:41:54.347824 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.347881 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:41:54.347934 9971 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 02:41:54.348122 9241 consensus_peers.cc:487] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 -> Peer 4080b99d797d41a09e92d1d747845838 (127.2.123.62:42551): Couldn't send request to peer 4080b99d797d41a09e92d1d747845838. Status: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:41:54.349464 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02f615b6adfd4a0eace185decbe5f9f6. Latest consensus state: current_term: 2 leader_uuid: "02f615b6adfd4a0eace185decbe5f9f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:54.349634 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:54.350755 9961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "02f615b6adfd4a0eace185decbe5f9f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:54.350874 9961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:54.351265 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "02f615b6adfd4a0eace185decbe5f9f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4080b99d797d41a09e92d1d747845838" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42551 } } peers { permanent_uuid: "02f615b6adfd4a0eace185decbe5f9f6" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 40139 } attrs { promote: false } } peers { permanent_uuid: "fe641e889deb47638ae4e4c23f4dbb29" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 42831 } attrs { promote: false } } }
I20260928 02:41:54.351384 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe641e889deb47638ae4e4c23f4dbb29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:41:54.351665 9975 mvcc.cc:205] Tried to move back new op lower bound from 7334147335562313728 to 7334147335422459904. Current Snapshot: MvccSnapshot[applied={T|T < 7334147335562313728 or (T in {7334147335562313728})}]
I20260928 02:41:54.360842 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.356722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35061 (local address 127.0.0.1:44540)
0928 02:41:54.357034 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.357048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:54.357127 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:54.357384 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:54.357386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:54.357399 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:54.357494 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.357500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.358401 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.358405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.359642 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.359652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.360459 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.360462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.360545 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.360589 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:41:54.360636 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:41:54.360706 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":34}
I20260928 02:41:54.361164 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.356466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41349 (local address 127.0.0.1:34014)
0928 02:41:54.356861 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.356872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:54.356944 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:54.358307 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:54.358310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:54.358327 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:54.358402 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.358408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.359094 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.359097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.359975 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.359984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.360792 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.360795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.360878 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.360900 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:41:54.360945 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:41:54.361029 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":61}
I20260928 02:41:54.361402 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.357206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.2.123.1:39239)
0928 02:41:54.357416 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:54.357420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:54.357434 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:41:54.357482 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:54.357484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:54.357530 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:54.357609 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:54.357941 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.358668 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.359998 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.360637 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.360739 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.360803 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:54.361215 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 02:41:54.361286 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260928 02:41:54.361490 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.356831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.2.123.3:35061)
0928 02:41:54.357138 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:54.357142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:54.357174 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:41:54.357226 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:54.357230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:54.357272 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:54.357374 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:54.357595 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.358305 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.359761 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.360341 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.361099 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.361159 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:54.361193 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 02:41:54.361358 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":73,"threads_started":1}
I20260928 02:41:54.361702 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.356594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.2.123.2:41349)
0928 02:41:54.358019 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:54.358023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:54.358100 (+ 77us) server_negotiation.cc:407] Connection header received
0928 02:41:54.358150 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:54.358153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:54.358197 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:54.358278 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:54.358491 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.358982 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.360072 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.360678 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.361350 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.361410 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:54.361445 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:41:54.361577 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":70,"threads_started":1}
I20260928 02:41:54.361166 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.356945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:39239 (local address 127.0.0.1:33832)
0928 02:41:54.357163 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.357175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:54.357252 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:54.357717 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:54.357719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:54.357735 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:54.357817 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.357823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.358787 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.358791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.359899 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.359905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.360849 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.360852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.360923 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.360942 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:41:54.360996 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:41:54.361049 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":22,"thread_start_us":92,"threads_started":1}
I20260928 02:41:54.362167 9826 tablet_service.cc:1467] Processing CreateTablet for tablet 5aec475ec7a5425d9070d011a3c70363 (DEFAULT_TABLE table=test-table [id=a7353fbd7b654f279ca69d9c4315b8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:54.362169 9689 tablet_service.cc:1467] Processing CreateTablet for tablet 5aec475ec7a5425d9070d011a3c70363 (DEFAULT_TABLE table=test-table [id=a7353fbd7b654f279ca69d9c4315b8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:54.362445 9826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aec475ec7a5425d9070d011a3c70363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:54.362445 9689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aec475ec7a5425d9070d011a3c70363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:54.364951 9984 tablet_bootstrap.cc:492] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: Bootstrap starting.
I20260928 02:41:54.365355 9552 tablet_service.cc:1467] Processing CreateTablet for tablet 5aec475ec7a5425d9070d011a3c70363 (DEFAULT_TABLE table=test-table [id=a7353fbd7b654f279ca69d9c4315b8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:54.365619 9552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aec475ec7a5425d9070d011a3c70363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:54.365872 9984 tablet_bootstrap.cc:654] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:54.366170 9983 tablet_bootstrap.cc:492] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: Bootstrap starting.
I20260928 02:41:54.366957 9983 tablet_bootstrap.cc:654] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:54.367125 9984 tablet_bootstrap.cc:492] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: No bootstrap required, opened a new log
I20260928 02:41:54.367614 9984 ts_tablet_manager.cc:1397] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:54.368093 9983 tablet_bootstrap.cc:492] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: No bootstrap required, opened a new log
I20260928 02:41:54.368185 9983 ts_tablet_manager.cc:1397] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:54.368264 9984 raft_consensus.cc:348] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.368371 9984 raft_consensus.cc:374] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:54.368405 9984 raft_consensus.cc:729] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb2f2879ec5457bab7a163cc9cb0059, State: Initialized, Role: FOLLOWER
I20260928 02:41:54.368352 9987 tablet_bootstrap.cc:492] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f: Bootstrap starting.
I20260928 02:41:54.368516 9984 consensus_queue.cc:260] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.368599 9983 raft_consensus.cc:348] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.368705 9983 raft_consensus.cc:374] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:54.368731 9983 raft_consensus.cc:729] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79410553cd9a4050b052449aa189151e, State: Initialized, Role: FOLLOWER
I20260928 02:41:54.368837 9983 consensus_queue.cc:260] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.368867 9984 ts_tablet_manager.cc:1428] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:54.369140 9983 ts_tablet_manager.cc:1428] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:54.369671 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.369270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.2:43387)
0928 02:41:54.369512 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.369574 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260928 02:41:54.370385 9987 tablet_bootstrap.cc:654] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:54.370627 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.370268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.3:53217)
0928 02:41:54.370473 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.370527 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260928 02:41:54.372001 9894 heartbeater.cc:503] Master 127.2.123.61:40139 was elected leader, sending a full tablet report...
I20260928 02:41:54.372777 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.372510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.2:36587)
0928 02:41:54.372664 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.372712 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 02:41:54.372922 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.371332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.3:51337)
0928 02:41:54.372819 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.372867 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1422}
I20260928 02:41:54.372972 9757 heartbeater.cc:503] Master 127.2.123.61:40139 was elected leader, sending a full tablet report...
I20260928 02:41:54.374235 9987 tablet_bootstrap.cc:492] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f: No bootstrap required, opened a new log
I20260928 02:41:54.374320 9987 ts_tablet_manager.cc:1397] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260928 02:41:54.374728 9987 raft_consensus.cc:348] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.374819 9987 raft_consensus.cc:374] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:54.374840 9987 raft_consensus.cc:729] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac7e976b8d44a7698eeb189152ba82f, State: Initialized, Role: FOLLOWER
I20260928 02:41:54.374946 9987 consensus_queue.cc:260] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.375248 9987 ts_tablet_manager.cc:1428] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:54.375877 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.375506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.1:52197)
0928 02:41:54.375733 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.375786 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
I20260928 02:41:54.376551 9620 heartbeater.cc:503] Master 127.2.123.61:40139 was elected leader, sending a full tablet report...
I20260928 02:41:54.376602 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.376242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.1:52811)
0928 02:41:54.376458 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.376511 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
W20260928 02:41:54.593813 9758 tablet.cc:2369] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:54.706404 9989 raft_consensus.cc:482] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:54.706548 9989 raft_consensus.cc:504] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.707687 9989 leader_election.cc:288] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdb2f2879ec5457bab7a163cc9cb0059 (127.2.123.2:41349), 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239)
I20260928 02:41:54.711329 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.708013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:46303 (local address 127.2.123.2:41349)
0928 02:41:54.708160 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:54.708164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:54.708450 (+ 286us) server_negotiation.cc:407] Connection header received
0928 02:41:54.708549 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:54.708552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:54.708593 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:54.708651 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:54.708861 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.709376 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.710452 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.710833 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.710920 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.710984 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:54.711148 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:41:54.711211 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:41:54.711369 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.708075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:34529 (local address 127.2.123.1:39239)
0928 02:41:54.708192 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:54.708196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:54.708215 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:41:54.708276 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:54.708280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:54.708344 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:54.708422 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:54.708777 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.709306 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.710446 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.710836 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.710925 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.710993 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:54.711191 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:41:54.711255 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:41:54.711369 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.707941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:39239 (local address 127.2.123.3:34529)
0928 02:41:54.708077 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.708093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:54.708195 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:54.708512 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:54.708516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:54.708534 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:54.708639 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.708644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.709404 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.709407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.710338 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.710347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.710952 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.710956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.711087 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.711104 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:41:54.711166 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:41:54.711233 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":58}
I20260928 02:41:54.711376 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.707941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41349 (local address 127.2.123.3:46303)
0928 02:41:54.708375 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.708385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:54.708460 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:54.708665 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:54.708667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:54.708682 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:54.708771 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.708776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.709467 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.709470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.710348 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:54.710356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.710935 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.710939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:54.711050 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.711068 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:41:54.711127 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:41:54.711192 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":213,"thread_start_us":125,"threads_started":1}
I20260928 02:41:54.712010 9572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aec475ec7a5425d9070d011a3c70363" candidate_uuid: "79410553cd9a4050b052449aa189151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac7e976b8d44a7698eeb189152ba82f" is_pre_election: true
I20260928 02:41:54.712007 9703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aec475ec7a5425d9070d011a3c70363" candidate_uuid: "79410553cd9a4050b052449aa189151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" is_pre_election: true
I20260928 02:41:54.712147 9572 raft_consensus.cc:2454] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79410553cd9a4050b052449aa189151e in term 0.
I20260928 02:41:54.712217 9703 raft_consensus.cc:2454] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79410553cd9a4050b052449aa189151e in term 0.
I20260928 02:41:54.712598 9779 leader_election.cc:302] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79410553cd9a4050b052449aa189151e, cdb2f2879ec5457bab7a163cc9cb0059; no voters:
I20260928 02:41:54.712836 9989 raft_consensus.cc:2781] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:54.712896 9989 raft_consensus.cc:482] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:54.712924 9989 raft_consensus.cc:3037] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:54.713893 9989 raft_consensus.cc:504] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.714227 9989 leader_election.cc:288] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [CANDIDATE]: Term 1 election: Requested vote from peers cdb2f2879ec5457bab7a163cc9cb0059 (127.2.123.2:41349), 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239)
I20260928 02:41:54.714551 9703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aec475ec7a5425d9070d011a3c70363" candidate_uuid: "79410553cd9a4050b052449aa189151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb2f2879ec5457bab7a163cc9cb0059"
I20260928 02:41:54.714675 9703 raft_consensus.cc:3037] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:54.714670 9572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aec475ec7a5425d9070d011a3c70363" candidate_uuid: "79410553cd9a4050b052449aa189151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac7e976b8d44a7698eeb189152ba82f"
I20260928 02:41:54.714802 9572 raft_consensus.cc:3037] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:54.715818 9572 raft_consensus.cc:2454] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79410553cd9a4050b052449aa189151e in term 1.
I20260928 02:41:54.715852 9703 raft_consensus.cc:2454] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79410553cd9a4050b052449aa189151e in term 1.
I20260928 02:41:54.716084 9780 leader_election.cc:302] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac7e976b8d44a7698eeb189152ba82f, 79410553cd9a4050b052449aa189151e; no voters:
I20260928 02:41:54.716267 9989 raft_consensus.cc:2781] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:54.716451 9989 raft_consensus.cc:686] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 1 LEADER]: Becoming Leader. State: Replica: 79410553cd9a4050b052449aa189151e, State: Running, Role: LEADER
I20260928 02:41:54.716682 9989 consensus_queue.cc:237] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } }
I20260928 02:41:54.717069 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.716834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.3:42775)
0928 02:41:54.716940 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.716995 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:41:54.718791 9247 catalog_manager.cc:5495] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e reported cstate change: term changed from 0 to 1, leader changed from <none> to 79410553cd9a4050b052449aa189151e (127.2.123.3). New cstate: current_term: 1 leader_uuid: "79410553cd9a4050b052449aa189151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79410553cd9a4050b052449aa189151e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:54.743708 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.743394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60414)
0928 02:41:54.743556 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.743624 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:41:54.744478 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.744282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.0.0.1:60418)
0928 02:41:54.744378 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.744423 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:41:54.811936 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.808764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.2.123.3:35061)
0928 02:41:54.808870 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:54.808873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:54.808886 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:41:54.809001 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:54.809003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:54.809059 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:54.809148 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:54.809310 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.809810 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.810632 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:54.810838 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:54.810931 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:54.811134 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:54.811578 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:54.811663 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:54.811764 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:41:54.811813 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:41:54.814680 9572 raft_consensus.cc:1261] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 1 FOLLOWER]: Refusing update from remote peer 79410553cd9a4050b052449aa189151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:54.815064 9703 raft_consensus.cc:1261] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 FOLLOWER]: Refusing update from remote peer 79410553cd9a4050b052449aa189151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:54.815209 9997 consensus_queue.cc:1035] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac7e976b8d44a7698eeb189152ba82f" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:54.815578 9989 consensus_queue.cc:1035] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdb2f2879ec5457bab7a163cc9cb0059" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:54.817155 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.816829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.1:59207)
0928 02:41:54.817000 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.817077 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 02:41:54.818538 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.818283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.2:45615)
0928 02:41:54.818398 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.818463 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 02:41:54.820806 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.820585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.3:45297)
0928 02:41:54.820686 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.820737 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:41:54.822223 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.822014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.2:56891)
0928 02:41:54.822108 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.822159 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:41:54.824812 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.824307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.3:50409)
0928 02:41:54.824690 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.824742 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319}
I20260928 02:41:54.826323 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:54.826003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42551 (local address 127.2.123.1:41211)
0928 02:41:54.826205 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.826255 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260928 02:41:54.828315 9255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:54.828505 9255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:54.831763 9255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02f615b6adfd4a0eace185decbe5f9f6: Sending DeleteTablet for 3 replicas of tablet 5aec475ec7a5425d9070d011a3c70363
I20260928 02:41:54.832454 9552 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aec475ec7a5425d9070d011a3c70363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:54 UTC) from {username='oryx', principal='oryx/127.2.123.61@KRBTEST.COM'} at 127.0.0.1:33832
I20260928 02:41:54.832669 9689 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aec475ec7a5425d9070d011a3c70363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:54 UTC) from {username='oryx', principal='oryx/127.2.123.61@KRBTEST.COM'} at 127.0.0.1:34014
I20260928 02:41:54.832870 10011 tablet_replica.cc:330] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f: stopping tablet replica
I20260928 02:41:54.832985 9826 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aec475ec7a5425d9070d011a3c70363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:54 UTC) from {username='oryx', principal='oryx/127.2.123.61@KRBTEST.COM'} at 127.0.0.1:44540
I20260928 02:41:54.833040 10012 tablet_replica.cc:330] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: stopping tablet replica
I20260928 02:41:54.833034 10011 raft_consensus.cc:2229] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:54.833134 10012 raft_consensus.cc:2229] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:54.833245 10012 raft_consensus.cc:2258] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:54.833271 10011 raft_consensus.cc:2258] T 5aec475ec7a5425d9070d011a3c70363 P 1ac7e976b8d44a7698eeb189152ba82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:54.833823 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 9399
I20260928 02:41:54.835644 10012 ts_tablet_manager.cc:1905] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:54.838392 10012 ts_tablet_manager.cc:1918] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:54.838476 10012 log.cc:1196] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/5aec475ec7a5425d9070d011a3c70363
I20260928 02:41:54.838739 10012 ts_tablet_manager.cc:1939] T 5aec475ec7a5425d9070d011a3c70363 P cdb2f2879ec5457bab7a163cc9cb0059: Deleting consensus metadata
I20260928 02:41:54.839669 9239 catalog_manager.cc:4841] TS cdb2f2879ec5457bab7a163cc9cb0059 (127.2.123.2:41349): tablet 5aec475ec7a5425d9070d011a3c70363 (table test-table [id=a7353fbd7b654f279ca69d9c4315b8ea]) successfully deleted
W20260928 02:41:54.840150 9981 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:54.839668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:39239 (local address 127.0.0.1:54086)
0928 02:41:54.839831 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:54.839845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:54.839930 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:54.840049 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:39239: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 02:41:54.840241 10013 tablet_replica.cc:330] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: stopping tablet replica
I20260928 02:41:54.840274 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 9626
I20260928 02:41:54.840358 10013 raft_consensus.cc:2229] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:54.841143 10013 raft_consensus.cc:2258] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:41:54.841602 9240 catalog_manager.cc:4584] TS 1ac7e976b8d44a7698eeb189152ba82f (127.2.123.1:39239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5aec475ec7a5425d9070d011a3c70363: Network error: Client connection negotiation failed: client connection to 127.2.123.1:39239: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 02:41:54.842188 10013 ts_tablet_manager.cc:1905] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:54.844246 10013 ts_tablet_manager.cc:1918] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:54.844313 10013 log.cc:1196] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/5aec475ec7a5425d9070d011a3c70363
I20260928 02:41:54.844539 10013 ts_tablet_manager.cc:1939] T 5aec475ec7a5425d9070d011a3c70363 P 79410553cd9a4050b052449aa189151e: Deleting consensus metadata
I20260928 02:41:54.845119 9239 catalog_manager.cc:4841] TS 79410553cd9a4050b052449aa189151e (127.2.123.3:35061): tablet 5aec475ec7a5425d9070d011a3c70363 (table test-table [id=a7353fbd7b654f279ca69d9c4315b8ea]) successfully deleted
I20260928 02:41:54.851620 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 9762
I20260928 02:41:54.856465 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 9119
I20260928 02:41:54.861095 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 9227
2026-09-28T02:41:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5562 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:54 dist-test-slave-rqvb krb5kdc[10016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:54 dist-test-slave-rqvb krb5kdc[10016](info): set up 2 sockets
Sep 28 02:41:54 dist-test-slave-rqvb krb5kdc[10016](info): commencing operation
krb5kdc: starting...
W20260928 02:41:55.822351 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:55 dist-test-slave-rqvb krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.62.keytab.
Entry for principal oryx/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.62.keytab.
I20260928 02:41:55.943037 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:35411
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.62.keytab
--principal=oryx/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:56.051082 10029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:56.051343 10029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:56.051383 10029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:56.055511 10029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:56.055574 10029 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:41:56.055589 10029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:56.055609 10029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:56.055625 10029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:56.055655 10029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:56.061022 10029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:35411
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.62.keytab
--principal=oryx/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:56.062112 10029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:56.063122 10029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:56.071538 10035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:56.071645 10034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:56.071549 10029 server_base.cc:1034] running on GCE node
W20260928 02:41:56.071553 10037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:56.072443 10029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:56.073531 10029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:56.074695 10029 hybrid_clock.cc:648] HybridClock initialized: now 1790563316074677 us; error 36 us; skew 500 ppm
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:56.077482 10029 init.cc:377] Logged in from keytab as oryx/127.2.123.62@KRBTEST.COM (short username oryx)
I20260928 02:41:56.078446 10029 webserver.cc:460] Webserver started at http://127.2.123.62:36791/ using document root <none> and password file <none>
I20260928 02:41:56.078980 10029 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:56.079042 10029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:56.079226 10029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:56.081497 10029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "35e1c121f6a143cd81758b96cdd9e7b9"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "5ad6dd1931f4a026123b7a3e7ba9976e"
server_key_iv: "ea0e9cacf63529eb6488abd9297aa5e7"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.082170 10029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "35e1c121f6a143cd81758b96cdd9e7b9"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "5ad6dd1931f4a026123b7a3e7ba9976e"
server_key_iv: "ea0e9cacf63529eb6488abd9297aa5e7"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.086200 10029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:41:56.088632 10044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:56.089928 10029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:56.090070 10029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "35e1c121f6a143cd81758b96cdd9e7b9"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "5ad6dd1931f4a026123b7a3e7ba9976e"
server_key_iv: "ea0e9cacf63529eb6488abd9297aa5e7"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.090174 10029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:56.103771 10029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:56.106745 10029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:56.106923 10029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:56.114714 10029 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:35411
I20260928 02:41:56.114786 10106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:35411 every 8 connection(s)
I20260928 02:41:56.115674 10029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:41:56.118602 10107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:56.118968 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 10029
I20260928 02:41:56.119091 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:41:56.119347 2540 external_mini_cluster.cc:1428] Setting key 70fcf7331bde8a0c381150145183bd44
I20260928 02:41:56.125411 10107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: Bootstrap starting.
I20260928 02:41:56.128126 10107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: Neither blocks nor log segments found. Creating new log.
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:56.128872 10107 log.cc:824] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:56.131363 10107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: No bootstrap required, opened a new log
I20260928 02:41:56.134804 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.120642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.2.123.62:35411)
0928 02:41:56.121031 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.121046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.121082 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:41:56.121748 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.121778 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.122154 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.122449 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.123458 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.124504 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.125612 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.126131 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.126246 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.129014 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:56.129021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:56.129036 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:56.129068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:56.131592 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.132059 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.132066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.132072 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.132219 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.132513 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.132516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.132518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.133008 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:56.133200 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.133499 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.133798 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20260928 02:41:56.135217 10107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e1c121f6a143cd81758b96cdd9e7b9" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35411 } }
I20260928 02:41:56.135479 10107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:56.135561 10107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e1c121f6a143cd81758b96cdd9e7b9, State: Initialized, Role: FOLLOWER
I20260928 02:41:56.136019 10107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e1c121f6a143cd81758b96cdd9e7b9" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35411 } }
I20260928 02:41:56.136154 10107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:41:56.136214 10107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:41:56.136307 10107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:56.137305 10107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e1c121f6a143cd81758b96cdd9e7b9" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35411 } }
I20260928 02:41:56.137621 10107 leader_election.cc:302] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e1c121f6a143cd81758b96cdd9e7b9; no voters:
I20260928 02:41:56.137914 10107 leader_election.cc:288] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:41:56.138046 10112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:56.138240 10112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [term 1 LEADER]: Becoming Leader. State: Replica: 35e1c121f6a143cd81758b96cdd9e7b9, State: Running, Role: LEADER
I20260928 02:41:56.138526 10112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e1c121f6a143cd81758b96cdd9e7b9" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35411 } }
I20260928 02:41:56.139099 10107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:41:56.140087 10113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35e1c121f6a143cd81758b96cdd9e7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e1c121f6a143cd81758b96cdd9e7b9" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35411 } } }
I20260928 02:41:56.140206 10113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:56.140477 10114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35e1c121f6a143cd81758b96cdd9e7b9. Latest consensus state: current_term: 1 leader_uuid: "35e1c121f6a143cd81758b96cdd9e7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e1c121f6a143cd81758b96cdd9e7b9" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35411 } } }
I20260928 02:41:56.140564 10114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:41:56.140980 10121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:41:56.144476 10121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:41:56.150578 10121 catalog_manager.cc:1347] Generated new cluster ID: 6a1be6430c7e40beb286a8b9436e3c3c
I20260928 02:41:56.150660 10121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:41:56.160511 10121 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:41:56.161460 10121 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:41:56.173347 10121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: Generated new TSK 0
I20260928 02:41:56.174247 10121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab.
Entry for principal oryx/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab.
I20260928 02:41:56.207127 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35411
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab
--principal=oryx/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:56.315568 10133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:56.315840 10133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:56.315883 10133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:56.319887 10133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:56.319952 10133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:56.319988 10133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:56.320025 10133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:41:56.325335 10133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab
--principal=oryx/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:56.326442 10133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:56.327458 10133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:56.337020 10139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:56.337110 10141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:56.337085 10133 server_base.cc:1034] running on GCE node
W20260928 02:41:56.337091 10138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:56.337610 10133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:56.338294 10133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:56.339432 10133 hybrid_clock.cc:648] HybridClock initialized: now 1790563316339384 us; error 56 us; skew 500 ppm
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:56.342427 10133 init.cc:377] Logged in from keytab as oryx/127.2.123.1@KRBTEST.COM (short username oryx)
I20260928 02:41:56.343317 10133 webserver.cc:460] Webserver started at http://127.2.123.1:44251/ using document root <none> and password file <none>
I20260928 02:41:56.343845 10133 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:56.343904 10133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:56.344110 10133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:56.346159 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "12c3a65b702a4ad0b62223c73bcf6c3a"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "1248b3618d9bd35e162aa90375b38334"
server_key_iv: "35c9dd0eb9448c29c3012c23a27d0f4d"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.346683 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "12c3a65b702a4ad0b62223c73bcf6c3a"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "1248b3618d9bd35e162aa90375b38334"
server_key_iv: "35c9dd0eb9448c29c3012c23a27d0f4d"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.350380 10133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:56.352777 10148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:56.354091 10133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:56.354182 10133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "12c3a65b702a4ad0b62223c73bcf6c3a"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "1248b3618d9bd35e162aa90375b38334"
server_key_iv: "35c9dd0eb9448c29c3012c23a27d0f4d"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.354285 10133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:56.380541 10133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:56.383711 10133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:56.383868 10133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:56.385211 10133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:56.385259 10133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:56.385298 10133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:56.385327 10133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:56.394841 10133 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:42007
I20260928 02:41:56.394908 10263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:42007 every 8 connection(s)
I20260928 02:41:56.395889 10133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:41:56.398487 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.386081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:36519 (local address 127.2.123.62:35411)
0928 02:41:56.386192 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.386197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.387242 (+ 1045us) server_negotiation.cc:407] Connection header received
0928 02:41:56.388365 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.388369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.388432 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.388529 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.390051 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.390886 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.392170 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.392399 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.392487 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.395132 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:56.395136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:56.395138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:56.395167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:56.396635 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.397099 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.397101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.397102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.397170 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.397457 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.397460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.397462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.397637 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:56.397716 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.398211 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.398327 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:41:56.399199 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.386609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35411 (local address 127.2.123.1:36519)
0928 02:41:56.387099 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.387141 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.388080 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.388780 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.388790 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.389195 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:56.389865 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.389885 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.391011 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.391014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.392058 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.392067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.392541 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.392545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.392711 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.393296 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:56.393316 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:56.394962 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:56.396732 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.396737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.396752 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.396993 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.397244 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.397247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.397249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.397369 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.397728 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:56.397734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:56.397952 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.398148 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.398444 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260928 02:41:56.400339 10264 heartbeater.cc:347] Connected to a master server at 127.2.123.62:35411
I20260928 02:41:56.400476 10264 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:56.400926 10264 heartbeater.cc:511] Master 127.2.123.62:35411 requested a full tablet report, sending...
I20260928 02:41:56.402484 10060 ts_manager.cc:194] Registered new tserver with Master: 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007)
I20260928 02:41:56.403846 10060 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:36519
I20260928 02:41:56.403888 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 10133
I20260928 02:41:56.403972 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:41:56.404248 2540 external_mini_cluster.cc:1428] Setting key 3862994ba7b1f9743c0083295f99a91e
I20260928 02:41:56.409832 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.402447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:58175 (local address 127.2.123.62:35411)
0928 02:41:56.402560 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.402564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.402693 (+ 129us) server_negotiation.cc:407] Connection header received
0928 02:41:56.402775 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.402778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.402818 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.402896 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.403342 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.404070 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.405142 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.405422 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.405561 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.406689 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:56.406692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:56.406695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:56.406718 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:56.408069 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.408556 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.408559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.408561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.408699 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.408978 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.408980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.408981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.409148 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:56.409224 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.409584 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.409718 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:41:56.409853 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.402353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35411 (local address 127.2.123.1:58175)
0928 02:41:56.402617 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.402629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.402715 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.402953 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.402955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.403130 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:56.403228 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.403233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.404216 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.404220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.405011 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.405021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.405680 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.405683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.405758 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.406237 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:56.406248 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:56.406555 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:56.408189 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.408193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.408195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.408439 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.408790 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.408793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.408795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.408894 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.409235 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:56.409238 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:56.409340 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.409464 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.409616 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab.
Entry for principal oryx/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab.
I20260928 02:41:56.431114 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35411
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab
--principal=oryx/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:56.540525 10273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:56.540817 10273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:56.540867 10273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:56.544899 10273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:56.544958 10273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:56.544996 10273 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:56.545040 10273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:41:56.550228 10273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab
--principal=oryx/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:56.551273 10273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:56.552263 10273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:56.561864 10273 server_base.cc:1034] running on GCE node
W20260928 02:41:56.561805 10279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:56.561789 10278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:56.561781 10281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:56.562458 10273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:56.563105 10273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:56.564281 10273 hybrid_clock.cc:648] HybridClock initialized: now 1790563316564264 us; error 30 us; skew 500 ppm
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:56.566969 10273 init.cc:377] Logged in from keytab as oryx/127.2.123.2@KRBTEST.COM (short username oryx)
I20260928 02:41:56.567853 10273 webserver.cc:460] Webserver started at http://127.2.123.2:46649/ using document root <none> and password file <none>
I20260928 02:41:56.568369 10273 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:56.568434 10273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:56.568620 10273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:56.570905 10273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "7bc859182ead47a29973b4dd95fafb2e"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "1dba0c8675132fffe993ec58054dbc2d"
server_key_iv: "b5d8b24c6d6790f651267291bf3a7d11"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.571477 10273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "7bc859182ead47a29973b4dd95fafb2e"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "1dba0c8675132fffe993ec58054dbc2d"
server_key_iv: "b5d8b24c6d6790f651267291bf3a7d11"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.575081 10273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:41:56.577530 10288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:56.578897 10273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:41:56.579007 10273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "7bc859182ead47a29973b4dd95fafb2e"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "1dba0c8675132fffe993ec58054dbc2d"
server_key_iv: "b5d8b24c6d6790f651267291bf3a7d11"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.579103 10273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:56.593621 10273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:56.596589 10273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:56.596764 10273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:56.598197 10273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:56.598246 10273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:56.598297 10273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:56.598325 10273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:56.608218 10273 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:33715
I20260928 02:41:56.608359 10403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:33715 every 8 connection(s)
I20260928 02:41:56.609249 10273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:41:56.611415 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.599172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:39039 (local address 127.2.123.62:35411)
0928 02:41:56.599318 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.599323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.600317 (+ 994us) server_negotiation.cc:407] Connection header received
0928 02:41:56.601516 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.601520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.601577 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.601684 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.603073 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.603705 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.604998 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.605217 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.605313 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.607955 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:56.607959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:56.607962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:56.607989 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:56.609509 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.610056 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.610059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.610061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.610157 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.610526 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.610528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.610529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.610709 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:56.610782 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.611208 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.611300 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:41:56.612202 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.599587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35411 (local address 127.2.123.2:39039)
0928 02:41:56.600154 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.600204 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.601283 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.601930 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.601940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.602401 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:56.602913 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.602927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.603845 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.603849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.604835 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.604846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.605383 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.605387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.605555 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.606129 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:56.606148 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:56.607789 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:56.609621 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.609627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.609642 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.609944 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.610247 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.610249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.610251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.610437 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.610783 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:56.610787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:56.610995 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.611158 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.611388 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":170,"threads_started":1}
I20260928 02:41:56.613471 10404 heartbeater.cc:347] Connected to a master server at 127.2.123.62:35411
I20260928 02:41:56.613615 10404 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:56.614157 10404 heartbeater.cc:511] Master 127.2.123.62:35411 requested a full tablet report, sending...
I20260928 02:41:56.615434 10061 ts_manager.cc:194] Registered new tserver with Master: 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715)
I20260928 02:41:56.616218 10061 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:39039
I20260928 02:41:56.617298 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 10273
I20260928 02:41:56.617417 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:41:56.617714 2540 external_mini_cluster.cc:1428] Setting key 379026ac5f3905d5c3b9c6722f679607
I20260928 02:41:56.623777 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.615763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:44999 (local address 127.2.123.62:35411)
0928 02:41:56.615878 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.615882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.616116 (+ 234us) server_negotiation.cc:407] Connection header received
0928 02:41:56.616212 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.616214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.616252 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.616324 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.616797 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.617320 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.618272 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.618525 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.618613 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.620547 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:56.620553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:56.620566 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:56.620617 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:56.622065 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.622523 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.622526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.622527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.622606 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.622891 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.622894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.622895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.623155 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:56.623241 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.623551 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.623659 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:41:56.623801 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.615710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35411 (local address 127.2.123.2:44999)
0928 02:41:56.616026 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.616040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.616135 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.616357 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.616360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.616574 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:56.616685 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.616692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.617464 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.617467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.618158 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.618169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.618643 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.618646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.618729 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.619854 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:56.619866 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:56.620328 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:56.622165 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.622169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.622171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.622406 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.622690 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.622693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.622695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.622784 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.623252 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:56.623255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:56.623362 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.623469 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.623615 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab.
Entry for principal oryx/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab.
I20260928 02:41:56.644435 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:35411
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab
--principal=oryx/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:56.751718 10413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:56.751983 10413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:56.752027 10413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:56.756176 10413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:56.756243 10413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:56.756282 10413 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:41:56.756325 10413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:41:56.761622 10413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab
--principal=oryx/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.123.62:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:56.762714 10413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:56.763760 10413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:41:56.773370 10419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:56.773343 10418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:56.773401 10421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:56.773641 10413 server_base.cc:1034] running on GCE node
I20260928 02:41:56.774124 10413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:56.774788 10413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:56.775971 10413 hybrid_clock.cc:648] HybridClock initialized: now 1790563316775938 us; error 47 us; skew 500 ppm
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:41:56.778676 10413 init.cc:377] Logged in from keytab as oryx/127.2.123.3@KRBTEST.COM (short username oryx)
I20260928 02:41:56.779564 10413 webserver.cc:460] Webserver started at http://127.2.123.3:41105/ using document root <none> and password file <none>
I20260928 02:41:56.780117 10413 fs_manager.cc:359] Metadata directory not provided
I20260928 02:41:56.780175 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:41:56.780360 10413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:41:56.782493 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "94d4f2a4945e4359bb6f74d6e43d5bca"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "d9722fffeba717345af5bae71ca4ddb1"
server_key_iv: "f44e2ea9504a06244b35cdd54a2706b2"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.783030 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "94d4f2a4945e4359bb6f74d6e43d5bca"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "d9722fffeba717345af5bae71ca4ddb1"
server_key_iv: "f44e2ea9504a06244b35cdd54a2706b2"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.786651 10413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:41:56.789043 10428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:56.790378 10413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:41:56.790473 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "94d4f2a4945e4359bb6f74d6e43d5bca"
format_stamp: "Formatted at 2026-09-28 02:41:56 on dist-test-slave-rqvb"
server_key: "d9722fffeba717345af5bae71ca4ddb1"
server_key_iv: "f44e2ea9504a06244b35cdd54a2706b2"
server_key_version: "encryptionkey@0"
I20260928 02:41:56.790573 10413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:41:56.804733 10413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:41:56.807878 10413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:41:56.808076 10413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:41:56.809578 10413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:41:56.809641 10413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:41:56.809690 10413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:41:56.809724 10413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:56.820951 10413 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:35845
I20260928 02:41:56.821051 10543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:35845 every 8 connection(s)
I20260928 02:41:56.822096 10413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:41:56.823827 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.810533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:56579 (local address 127.2.123.62:35411)
0928 02:41:56.810656 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.810660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.811649 (+ 989us) server_negotiation.cc:407] Connection header received
0928 02:41:56.812569 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.812573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.812634 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.812735 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.814129 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.814931 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.816240 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.816577 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.816713 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.820113 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:56.820116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:56.820118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:56.820144 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:56.821733 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.822304 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.822308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.822309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.822416 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.822795 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.822799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.822800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.823023 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:56.823102 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.823602 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.823703 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:41:56.824545 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.811013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35411 (local address 127.2.123.3:56579)
0928 02:41:56.811494 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.811537 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.812335 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.812919 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.812928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.813357 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:56.813930 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.813946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.815064 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.815068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.816133 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.816143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.816775 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.816779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.816930 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.817491 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:56.817517 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:56.819945 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:56.821871 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.821875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.821886 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.822211 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.822516 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.822519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.822521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.822681 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.823124 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:56.823129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:56.823354 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.823523 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.823764 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":124,"threads_started":1}
I20260928 02:41:56.825650 10544 heartbeater.cc:347] Connected to a master server at 127.2.123.62:35411
I20260928 02:41:56.825794 10544 heartbeater.cc:464] Registering TS with master...
I20260928 02:41:56.826300 10544 heartbeater.cc:511] Master 127.2.123.62:35411 requested a full tablet report, sending...
I20260928 02:41:56.827286 10060 ts_manager.cc:194] Registered new tserver with Master: 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845)
I20260928 02:41:56.828073 10060 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:56579
I20260928 02:41:56.830617 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 10413
I20260928 02:41:56.830734 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:41:56.831108 2540 external_mini_cluster.cc:1428] Setting key f35805d5c18d3d1e70df90cd368ef79b
I20260928 02:41:56.833161 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:41:56.834424 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.827724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:33729 (local address 127.2.123.62:35411)
0928 02:41:56.827836 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.827840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.827987 (+ 147us) server_negotiation.cc:407] Connection header received
0928 02:41:56.828092 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.828094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.828127 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.828195 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.828626 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.829114 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.829898 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.830175 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.830271 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.831336 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:56.831338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:56.831340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:56.831424 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:56.832903 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.833326 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.833328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.833329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.833400 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.833659 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.833661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.833662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.833805 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:56.833897 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.834213 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.834301 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:41:56.834481 10550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.827662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35411 (local address 127.2.123.3:33729)
0928 02:41:56.827915 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.827928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.828017 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.828222 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.828225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.828424 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:56.828531 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.828537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.829228 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.829231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.829798 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.829805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.830270 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.830274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.830361 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.830811 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:56.830825 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:56.831221 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:56.833003 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.833006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.833009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.833248 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.833478 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:56.833481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:56.833483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:56.833582 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:56.833899 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:56.833902 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:56.834026 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.834172 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.834303 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:41:56.856221 10110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:56.850211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.2.123.62:35411)
0928 02:41:56.850343 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.850347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.850411 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:41:56.850452 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.850454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.850494 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.850559 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.851249 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.851821 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.852741 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.852956 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.853034 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.856068 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33076: BlockingRecv error: recv got EOF from 127.0.0.1:33076 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 02:41:56 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:56.865589 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.857683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.2.123.62:35411)
0928 02:41:56.857790 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.857794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.857934 (+ 140us) server_negotiation.cc:407] Connection header received
0928 02:41:56.858092 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.858095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.858140 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.858196 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:56.858775 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.859276 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.860069 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.860268 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.860348 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.862616 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:56.862620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:56.862622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:56.862646 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:56.863982 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.864394 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.864397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.864399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.864487 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:56.864761 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:56.864764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:56.864766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:56.864952 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:56.865058 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.865319 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.865433 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:41:56.868772 10060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:56.871846 10060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:56.886113 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.880487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35845 (local address 127.0.0.1:56366)
0928 02:41:56.881441 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.881500 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.881626 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.882262 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.882268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.882289 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:56.882412 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.882418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.883573 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.883576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.884762 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.884771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.885550 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.885553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.885658 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.885699 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.885808 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.885934 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260928 02:41:56.886123 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.881210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.2.123.3:35845)
0928 02:41:56.881602 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.881618 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.881641 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:41:56.881794 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.881800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.882083 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.882262 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:56.882531 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.883466 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.884886 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.885438 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.885543 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.885714 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.885846 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.885951 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":91,"threads_started":1}
I20260928 02:41:56.886863 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.881099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:42007 (local address 127.0.0.1:48282)
0928 02:41:56.881843 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.881865 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.882006 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.882489 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.882492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.882509 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:56.882616 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.882621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.883970 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.883973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.885317 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.885336 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.886564 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.886566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.886637 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.886654 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.886711 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.886763 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":86,"thread_start_us":91,"threads_started":1}
I20260928 02:41:56.887117 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.881456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.2.123.1:42007)
0928 02:41:56.881787 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.881809 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.881943 (+ 134us) server_negotiation.cc:407] Connection header received
0928 02:41:56.882118 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.882123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.882366 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.882501 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:56.882705 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.883868 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.885455 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.886259 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.886364 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.886514 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.886774 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.886903 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":64,"threads_started":1}
I20260928 02:41:56.887771 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.880408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33715 (local address 127.0.0.1:56262)
0928 02:41:56.881615 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:56.881629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:56.881718 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:56.882408 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:56.882411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:56.882428 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:56.882529 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.882535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.883967 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.883979 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.885322 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:56.885330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.887426 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.887429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:56.887513 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.887535 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:41:56.887590 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:41:56.887653 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":90,"thread_start_us":86,"threads_started":1}
I20260928 02:41:56.887841 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:56.880593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.2.123.2:33715)
0928 02:41:56.881199 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:56.881212 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:56.881698 (+ 486us) server_negotiation.cc:407] Connection header received
0928 02:41:56.881866 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:56.881871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:56.882080 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:56.882215 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:56.882659 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.883834 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.885444 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:56.887300 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:56.887414 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:56.887570 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:56.887621 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:41:56.887705 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":116,"threads_started":1}
I20260928 02:41:56.889437 10478 tablet_service.cc:1467] Processing CreateTablet for tablet f7830e9c9d07490a9bcaac67266f4fbb (DEFAULT_TABLE table=test-table [id=94380613d2e84ee795b750a2c2eec2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:56.890044 10338 tablet_service.cc:1467] Processing CreateTablet for tablet f7830e9c9d07490a9bcaac67266f4fbb (DEFAULT_TABLE table=test-table [id=94380613d2e84ee795b750a2c2eec2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:56.890033 10198 tablet_service.cc:1467] Processing CreateTablet for tablet f7830e9c9d07490a9bcaac67266f4fbb (DEFAULT_TABLE table=test-table [id=94380613d2e84ee795b750a2c2eec2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:56.890617 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7830e9c9d07490a9bcaac67266f4fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:56.890918 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7830e9c9d07490a9bcaac67266f4fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:56.890959 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7830e9c9d07490a9bcaac67266f4fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:56.897801 10572 tablet_bootstrap.cc:492] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: Bootstrap starting.
I20260928 02:41:56.898970 10573 tablet_bootstrap.cc:492] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Bootstrap starting.
I20260928 02:41:56.899916 10572 tablet_bootstrap.cc:654] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:56.900673 10572 log.cc:824] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:56.900779 10573 tablet_bootstrap.cc:654] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:56.901543 10573 log.cc:824] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:56.902204 10574 tablet_bootstrap.cc:492] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: Bootstrap starting.
I20260928 02:41:56.902817 10572 tablet_bootstrap.cc:492] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: No bootstrap required, opened a new log
I20260928 02:41:56.903019 10572 ts_tablet_manager.cc:1397] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:41:56.903477 10573 tablet_bootstrap.cc:492] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: No bootstrap required, opened a new log
I20260928 02:41:56.903649 10573 ts_tablet_manager.cc:1397] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:41:56.904274 10574 tablet_bootstrap.cc:654] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:56.905074 10574 log.cc:824] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:41:56.906144 10572 raft_consensus.cc:348] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:56.906358 10572 raft_consensus.cc:374] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:56.906402 10572 raft_consensus.cc:729] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc859182ead47a29973b4dd95fafb2e, State: Initialized, Role: FOLLOWER
I20260928 02:41:56.906811 10572 consensus_queue.cc:260] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:56.907047 10574 tablet_bootstrap.cc:492] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: No bootstrap required, opened a new log
I20260928 02:41:56.907217 10574 ts_tablet_manager.cc:1397] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:41:56.907229 10573 raft_consensus.cc:348] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:56.907469 10573 raft_consensus.cc:374] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:56.907497 10404 heartbeater.cc:503] Master 127.2.123.62:35411 was elected leader, sending a full tablet report...
I20260928 02:41:56.907516 10573 raft_consensus.cc:729] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4f2a4945e4359bb6f74d6e43d5bca, State: Initialized, Role: FOLLOWER
I20260928 02:41:56.907760 10572 ts_tablet_manager.cc:1428] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:41:56.907943 10573 consensus_queue.cc:260] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:56.908555 10544 heartbeater.cc:503] Master 127.2.123.62:35411 was elected leader, sending a full tablet report...
I20260928 02:41:56.908833 10573 ts_tablet_manager.cc:1428] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260928 02:41:56.910676 10574 raft_consensus.cc:348] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:56.910904 10574 raft_consensus.cc:374] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:56.910964 10574 raft_consensus.cc:729] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c3a65b702a4ad0b62223c73bcf6c3a, State: Initialized, Role: FOLLOWER
I20260928 02:41:56.911357 10574 consensus_queue.cc:260] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:56.912141 10574 ts_tablet_manager.cc:1428] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:41:56.912451 10264 heartbeater.cc:503] Master 127.2.123.62:35411 was elected leader, sending a full tablet report...
I20260928 02:41:57.044345 10578 raft_consensus.cc:482] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:57.044543 10578 raft_consensus.cc:504] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:57.045655 10578 leader_election.cc:288] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845), 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007)
I20260928 02:41:57.050139 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.045894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35845 (local address 127.2.123.2:33871)
0928 02:41:57.046108 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.046123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.046230 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.046496 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.046498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.046511 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:57.046628 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.046634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.047678 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.047682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.048964 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.048975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.049739 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.049742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.049825 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.049845 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.049902 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.049988 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260928 02:41:57.050159 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.046028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:33871 (local address 127.2.123.3:35845)
0928 02:41:57.046131 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.046135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.046217 (+ 82us) server_negotiation.cc:407] Connection header received
0928 02:41:57.046339 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.046342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.046389 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.046471 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:57.046740 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.047532 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.049094 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.049626 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.049735 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.049798 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.049935 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.050034 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:41:57.050369 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.045987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:42007 (local address 127.2.123.2:59541)
0928 02:41:57.046321 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.046334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.046412 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.046619 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.046622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.046634 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:57.046731 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.046737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.047692 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.047695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.049208 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.049215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.050010 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.050013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.050100 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.050119 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.050190 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.050247 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":99,"threads_started":1}
I20260928 02:41:57.050479 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.046041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:59541 (local address 127.2.123.1:42007)
0928 02:41:57.046166 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.046171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.046408 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:41:57.046482 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.046484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.046530 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.046597 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:57.046826 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.047593 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.049323 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.049878 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.050000 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.050061 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.050277 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.050359 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:41:57.050851 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7830e9c9d07490a9bcaac67266f4fbb" candidate_uuid: "7bc859182ead47a29973b4dd95fafb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" is_pre_election: true
I20260928 02:41:57.050976 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7830e9c9d07490a9bcaac67266f4fbb" candidate_uuid: "7bc859182ead47a29973b4dd95fafb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" is_pre_election: true
I20260928 02:41:57.051172 10498 raft_consensus.cc:2454] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc859182ead47a29973b4dd95fafb2e in term 0.
I20260928 02:41:57.051208 10218 raft_consensus.cc:2454] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc859182ead47a29973b4dd95fafb2e in term 0.
I20260928 02:41:57.051673 10289 leader_election.cc:302] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bc859182ead47a29973b4dd95fafb2e, 94d4f2a4945e4359bb6f74d6e43d5bca; no voters:
I20260928 02:41:57.051941 10578 raft_consensus.cc:2781] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:57.052024 10578 raft_consensus.cc:482] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:57.052055 10578 raft_consensus.cc:3037] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.053119 10578 raft_consensus.cc:504] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:57.053447 10578 leader_election.cc:288] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [CANDIDATE]: Term 1 election: Requested vote from peers 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845), 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007)
I20260928 02:41:57.053754 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7830e9c9d07490a9bcaac67266f4fbb" candidate_uuid: "7bc859182ead47a29973b4dd95fafb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca"
I20260928 02:41:57.053761 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7830e9c9d07490a9bcaac67266f4fbb" candidate_uuid: "7bc859182ead47a29973b4dd95fafb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a"
I20260928 02:41:57.053853 10498 raft_consensus.cc:3037] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.053896 10218 raft_consensus.cc:3037] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.054981 10218 raft_consensus.cc:2454] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bc859182ead47a29973b4dd95fafb2e in term 1.
I20260928 02:41:57.055290 10291 leader_election.cc:302] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 7bc859182ead47a29973b4dd95fafb2e; no voters:
I20260928 02:41:57.055297 10498 raft_consensus.cc:2454] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bc859182ead47a29973b4dd95fafb2e in term 1.
I20260928 02:41:57.055445 10578 raft_consensus.cc:2781] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:57.055663 10578 raft_consensus.cc:686] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 1 LEADER]: Becoming Leader. State: Replica: 7bc859182ead47a29973b4dd95fafb2e, State: Running, Role: LEADER
I20260928 02:41:57.055986 10578 consensus_queue.cc:237] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } }
I20260928 02:41:57.059051 10060 catalog_manager.cc:5495] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2). New cstate: current_term: 1 leader_uuid: "7bc859182ead47a29973b4dd95fafb2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } health_report { overall_health: UNKNOWN } } }
W20260928 02:41:57.073091 10545 tablet.cc:2369] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:57.076712 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.072927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.2.123.2:33715)
0928 02:41:57.073044 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.073047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.073060 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:41:57.073233 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.073236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.073334 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.073436 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:57.073724 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.074399 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.075311 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.075537 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.075619 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.075880 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.076314 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.076397 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.076510 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.076560 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:41:57.082305 10218 raft_consensus.cc:1261] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Refusing update from remote peer 7bc859182ead47a29973b4dd95fafb2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.082342 10498 raft_consensus.cc:1261] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Refusing update from remote peer 7bc859182ead47a29973b4dd95fafb2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.082991 10578 consensus_queue.cc:1035] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.083226 10582 consensus_queue.cc:1035] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.091873 10586 mvcc.cc:205] Tried to move back new op lower bound from 7334147346762481664 to 7334147346664677376. Current Snapshot: MvccSnapshot[applied={T|T < 7334147346762481664}]
I20260928 02:41:57.094769 10585 mvcc.cc:205] Tried to move back new op lower bound from 7334147346762481664 to 7334147346664677376. Current Snapshot: MvccSnapshot[applied={T|T < 7334147346762481664 or (T in {7334147346762481664})}]
I20260928 02:41:57.099677 10060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:57.099859 10060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:57.102280 10060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: Sending DeleteTablet for 3 replicas of tablet f7830e9c9d07490a9bcaac67266f4fbb
I20260928 02:41:57.103035 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet f7830e9c9d07490a9bcaac67266f4fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56262
I20260928 02:41:57.103044 10198 tablet_service.cc:1514] Processing DeleteTablet for tablet f7830e9c9d07490a9bcaac67266f4fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:48282
I20260928 02:41:57.103080 10478 tablet_service.cc:1514] Processing DeleteTablet for tablet f7830e9c9d07490a9bcaac67266f4fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56366
I20260928 02:41:57.103605 10594 tablet_replica.cc:330] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: stopping tablet replica
I20260928 02:41:57.103737 10595 tablet_replica.cc:330] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: stopping tablet replica
I20260928 02:41:57.103940 10594 raft_consensus.cc:2229] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:57.103973 10595 raft_consensus.cc:2229] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:57.104247 10595 raft_consensus.cc:2258] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.104247 10594 raft_consensus.cc:2258] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.104777 10060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:57.105192 10060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:57.105777 10594 ts_tablet_manager.cc:1905] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.104597 10596 tablet_replica.cc:330] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: stopping tablet replica
I20260928 02:41:57.106745 10596 raft_consensus.cc:2229] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:57.107151 10596 raft_consensus.cc:2258] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.108544 10594 ts_tablet_manager.cc:1918] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:57.108650 10594 log.cc:1196] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/f7830e9c9d07490a9bcaac67266f4fbb
I20260928 02:41:57.108950 10594 ts_tablet_manager.cc:1939] T f7830e9c9d07490a9bcaac67266f4fbb P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting consensus metadata
I20260928 02:41:57.108951 10595 ts_tablet_manager.cc:1905] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.110392 10198 tablet_service.cc:1467] Processing CreateTablet for tablet 8688726fe41f49349f995bb08623997f (DEFAULT_TABLE table=test-table [id=1b4bcce992ee44ab83fcf59d8475f768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.110457 10047 catalog_manager.cc:4841] TS 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007): tablet f7830e9c9d07490a9bcaac67266f4fbb (table test-table [id=94380613d2e84ee795b750a2c2eec2cf]) successfully deleted
I20260928 02:41:57.110648 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8688726fe41f49349f995bb08623997f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.111605 10478 tablet_service.cc:1467] Processing CreateTablet for tablet 8688726fe41f49349f995bb08623997f (DEFAULT_TABLE table=test-table [id=1b4bcce992ee44ab83fcf59d8475f768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.111865 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8688726fe41f49349f995bb08623997f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.112195 10595 ts_tablet_manager.cc:1918] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:57.112330 10595 log.cc:1196] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/f7830e9c9d07490a9bcaac67266f4fbb
I20260928 02:41:57.112661 10595 ts_tablet_manager.cc:1939] T f7830e9c9d07490a9bcaac67266f4fbb P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting consensus metadata
I20260928 02:41:57.112964 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 8688726fe41f49349f995bb08623997f (DEFAULT_TABLE table=test-table [id=1b4bcce992ee44ab83fcf59d8475f768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.113237 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8688726fe41f49349f995bb08623997f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.113817 10573 tablet_bootstrap.cc:492] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: Bootstrap starting.
I20260928 02:41:57.114322 10045 catalog_manager.cc:4841] TS 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845): tablet f7830e9c9d07490a9bcaac67266f4fbb (table test-table [id=94380613d2e84ee795b750a2c2eec2cf]) successfully deleted
I20260928 02:41:57.114766 10573 tablet_bootstrap.cc:654] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.115438 10572 tablet_bootstrap.cc:492] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: Bootstrap starting.
I20260928 02:41:57.115507 10574 tablet_bootstrap.cc:492] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: Bootstrap starting.
I20260928 02:41:57.115803 10573 tablet_bootstrap.cc:492] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: No bootstrap required, opened a new log
I20260928 02:41:57.115871 10573 ts_tablet_manager.cc:1397] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:57.116312 10572 tablet_bootstrap.cc:654] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.116281 10573 raft_consensus.cc:348] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.116401 10573 raft_consensus.cc:374] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.116433 10573 raft_consensus.cc:729] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4f2a4945e4359bb6f74d6e43d5bca, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.116454 10574 tablet_bootstrap.cc:654] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.116550 10573 consensus_queue.cc:260] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.116799 10573 ts_tablet_manager.cc:1428] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:57.117746 10574 tablet_bootstrap.cc:492] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: No bootstrap required, opened a new log
I20260928 02:41:57.117826 10574 ts_tablet_manager.cc:1397] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:57.118229 10572 tablet_bootstrap.cc:492] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: No bootstrap required, opened a new log
I20260928 02:41:57.118314 10572 ts_tablet_manager.cc:1397] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:41:57.118538 10574 raft_consensus.cc:348] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.118637 10574 raft_consensus.cc:374] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.118662 10574 raft_consensus.cc:729] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c3a65b702a4ad0b62223c73bcf6c3a, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.118722 10572 raft_consensus.cc:348] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.118770 10574 consensus_queue.cc:260] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.118818 10572 raft_consensus.cc:374] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.118849 10572 raft_consensus.cc:729] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc859182ead47a29973b4dd95fafb2e, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.118964 10572 consensus_queue.cc:260] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.119204 10572 ts_tablet_manager.cc:1428] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:57.119640 10574 ts_tablet_manager.cc:1428] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:57.122396 10596 ts_tablet_manager.cc:1905] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.125003 10596 ts_tablet_manager.cc:1918] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:57.125095 10596 log.cc:1196] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/f7830e9c9d07490a9bcaac67266f4fbb
I20260928 02:41:57.125360 10596 ts_tablet_manager.cc:1939] T f7830e9c9d07490a9bcaac67266f4fbb P 7bc859182ead47a29973b4dd95fafb2e: Deleting consensus metadata
I20260928 02:41:57.126437 10048 catalog_manager.cc:4841] TS 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715): tablet f7830e9c9d07490a9bcaac67266f4fbb (table test-table [id=94380613d2e84ee795b750a2c2eec2cf]) successfully deleted
W20260928 02:41:57.147161 10265 tablet.cc:2369] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:41:57.360494 10405 tablet.cc:2369] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:41:57.409148 10579 raft_consensus.cc:482] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:57.409324 10579 raft_consensus.cc:504] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.410581 10579 leader_election.cc:288] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007), 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715)
I20260928 02:41:57.414471 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.410899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:48393 (local address 127.2.123.2:33715)
0928 02:41:57.411037 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.411043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.411296 (+ 253us) server_negotiation.cc:407] Connection header received
0928 02:41:57.411412 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.411426 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.411471 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.411542 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:57.411778 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.412316 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.413424 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.413876 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.413993 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.414066 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.414287 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.414350 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:41:57.414505 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.410784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:42007 (local address 127.2.123.3:43727)
0928 02:41:57.411105 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.411119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.411212 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.411434 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.411437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.411450 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:57.411581 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.411587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.412330 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.412333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.413311 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.413319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.414011 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.414015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.414161 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.414183 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.414251 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.414338 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":52,"thread_start_us":66,"threads_started":1}
I20260928 02:41:57.414541 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.410897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:43727 (local address 127.2.123.1:42007)
0928 02:41:57.411030 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.411036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.411196 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:41:57.411295 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.411298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.411346 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.411426 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:57.411684 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.412226 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.413424 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.413880 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.414008 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.414076 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.414307 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.414403 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:41:57.414732 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.410784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33715 (local address 127.2.123.3:48393)
0928 02:41:57.411196 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.411212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.411318 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.411565 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.411567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.411581 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:57.411679 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.411686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.412423 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.412426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.413309 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.413318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.414017 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.414022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.414170 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.414195 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.414274 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.414352 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":216,"thread_start_us":77,"threads_started":1}
I20260928 02:41:57.415015 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8688726fe41f49349f995bb08623997f" candidate_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" is_pre_election: true
I20260928 02:41:57.415149 10218 raft_consensus.cc:2454] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d4f2a4945e4359bb6f74d6e43d5bca in term 0.
I20260928 02:41:57.415475 10431 leader_election.cc:302] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 94d4f2a4945e4359bb6f74d6e43d5bca; no voters:
I20260928 02:41:57.415709 10579 raft_consensus.cc:2781] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:57.415766 10579 raft_consensus.cc:482] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:57.415792 10579 raft_consensus.cc:3037] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.415714 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8688726fe41f49349f995bb08623997f" candidate_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc859182ead47a29973b4dd95fafb2e" is_pre_election: true
I20260928 02:41:57.415930 10358 raft_consensus.cc:2454] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d4f2a4945e4359bb6f74d6e43d5bca in term 0.
I20260928 02:41:57.416834 10579 raft_consensus.cc:504] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.417171 10579 leader_election.cc:288] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [CANDIDATE]: Term 1 election: Requested vote from peers 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007), 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715)
I20260928 02:41:57.417465 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8688726fe41f49349f995bb08623997f" candidate_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a"
I20260928 02:41:57.417505 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8688726fe41f49349f995bb08623997f" candidate_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc859182ead47a29973b4dd95fafb2e"
I20260928 02:41:57.417590 10218 raft_consensus.cc:3037] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.417635 10358 raft_consensus.cc:3037] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.418489 10218 raft_consensus.cc:2454] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d4f2a4945e4359bb6f74d6e43d5bca in term 1.
I20260928 02:41:57.418598 10358 raft_consensus.cc:2454] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d4f2a4945e4359bb6f74d6e43d5bca in term 1.
I20260928 02:41:57.418744 10431 leader_election.cc:302] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 94d4f2a4945e4359bb6f74d6e43d5bca; no voters:
I20260928 02:41:57.418910 10579 raft_consensus.cc:2781] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:57.419129 10579 raft_consensus.cc:686] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 LEADER]: Becoming Leader. State: Replica: 94d4f2a4945e4359bb6f74d6e43d5bca, State: Running, Role: LEADER
I20260928 02:41:57.419358 10579 consensus_queue.cc:237] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.421414 10061 catalog_manager.cc:5495] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca reported cstate change: term changed from 0 to 1, leader changed from <none> to 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3). New cstate: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } health_report { overall_health: HEALTHY } } }
Sep 28 02:41:57 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563316, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.62@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:57.519927 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.511596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:35411 (local address 127.0.0.1:33100)
0928 02:41:57.511943 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.511958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.512085 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.512311 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.512314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.512743 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:41:57.512837 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.512843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.513644 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.513647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.514279 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.514289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.514744 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.514747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.514820 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.515349 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:41:57.515360 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:41:57.516757 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:41:57.518461 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:57.518470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:57.518477 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:57.518696 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:57.518928 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:41:57.518931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:41:57.518933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:41:57.519010 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:41:57.519348 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:41:57.519357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:41:57.519461 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.519554 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.519710 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":129,"threads_started":1}
I20260928 02:41:57.519927 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.511696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.2.123.62:35411)
0928 02:41:57.511925 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.511935 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.512019 (+ 84us) server_negotiation.cc:407] Connection header received
0928 02:41:57.512173 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.512176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.512212 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.512283 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:57.512953 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.513540 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.514385 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.514644 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.514730 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.516878 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:57.516882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:57.516885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:57.516908 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:57.518364 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:57.518779 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:57.518782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:57.518784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:57.518854 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:57.519082 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:57.519085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:57.519087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:57.519254 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:57.519336 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.519639 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.519764 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260928 02:41:57.523978 10061 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:33100:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:41:57.524641 10061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:57.529237 10198 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7ae49066ab48e49e3045b99c861bcc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22a6cb3a4e67421fb998bd1400959b62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:57.529242 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7ae49066ab48e49e3045b99c861bcc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22a6cb3a4e67421fb998bd1400959b62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:57.529242 10478 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7ae49066ab48e49e3045b99c861bcc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22a6cb3a4e67421fb998bd1400959b62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:41:57.529610 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7ae49066ab48e49e3045b99c861bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.529610 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7ae49066ab48e49e3045b99c861bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.529613 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7ae49066ab48e49e3045b99c861bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.532011 10574 tablet_bootstrap.cc:492] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: Bootstrap starting.
I20260928 02:41:57.532239 10572 tablet_bootstrap.cc:492] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: Bootstrap starting.
I20260928 02:41:57.532395 10573 tablet_bootstrap.cc:492] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: Bootstrap starting.
I20260928 02:41:57.532961 10574 tablet_bootstrap.cc:654] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.533020 10572 tablet_bootstrap.cc:654] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.533304 10573 tablet_bootstrap.cc:654] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.534015 10572 tablet_bootstrap.cc:492] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: No bootstrap required, opened a new log
I20260928 02:41:57.534102 10572 ts_tablet_manager.cc:1397] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:41:57.534483 10572 raft_consensus.cc:348] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.534610 10572 raft_consensus.cc:374] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.534632 10574 tablet_bootstrap.cc:492] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: No bootstrap required, opened a new log
I20260928 02:41:57.534648 10572 raft_consensus.cc:729] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc859182ead47a29973b4dd95fafb2e, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.534660 10573 tablet_bootstrap.cc:492] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: No bootstrap required, opened a new log
I20260928 02:41:57.534703 10574 ts_tablet_manager.cc:1397] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:41:57.534727 10573 ts_tablet_manager.cc:1397] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:41:57.534763 10572 consensus_queue.cc:260] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.535012 10572 ts_tablet_manager.cc:1428] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:57.535039 10582 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.535104 10574 raft_consensus.cc:348] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.535171 10582 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:57.535125 10573 raft_consensus.cc:348] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.535192 10574 raft_consensus.cc:374] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.535210 10573 raft_consensus.cc:374] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.535220 10574 raft_consensus.cc:729] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c3a65b702a4ad0b62223c73bcf6c3a, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.535230 10573 raft_consensus.cc:729] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4f2a4945e4359bb6f74d6e43d5bca, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.535324 10573 consensus_queue.cc:260] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.535331 10574 consensus_queue.cc:260] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.535555 10573 ts_tablet_manager.cc:1428] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:57.535560 10574 ts_tablet_manager.cc:1428] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:57.535518 10579 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.535622 10579 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:57.535745 10580 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.535862 10580 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:57.537724 10579 raft_consensus.cc:482] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:57.537796 10579 raft_consensus.cc:504] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.538134 10579 leader_election.cc:288] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007), 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715)
I20260928 02:41:57.538434 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7ae49066ab48e49e3045b99c861bcc" candidate_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" is_pre_election: true
I20260928 02:41:57.538540 10218 raft_consensus.cc:2454] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d4f2a4945e4359bb6f74d6e43d5bca in term 0.
I20260928 02:41:57.538792 10431 leader_election.cc:302] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 94d4f2a4945e4359bb6f74d6e43d5bca; no voters:
I20260928 02:41:57.538776 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7ae49066ab48e49e3045b99c861bcc" candidate_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc859182ead47a29973b4dd95fafb2e" is_pre_election: true
I20260928 02:41:57.538898 10358 raft_consensus.cc:2454] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d4f2a4945e4359bb6f74d6e43d5bca in term 0.
I20260928 02:41:57.538940 10579 raft_consensus.cc:2781] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:57.538993 10579 raft_consensus.cc:482] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:57.539034 10579 raft_consensus.cc:3037] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.539921 10579 raft_consensus.cc:504] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.540226 10579 leader_election.cc:288] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [CANDIDATE]: Term 1 election: Requested vote from peers 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007), 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715)
I20260928 02:41:57.540601 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7ae49066ab48e49e3045b99c861bcc" candidate_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc859182ead47a29973b4dd95fafb2e"
I20260928 02:41:57.540710 10358 raft_consensus.cc:3037] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.540678 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7ae49066ab48e49e3045b99c861bcc" candidate_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a"
I20260928 02:41:57.540783 10218 raft_consensus.cc:3037] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.541612 10358 raft_consensus.cc:2454] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d4f2a4945e4359bb6f74d6e43d5bca in term 1.
I20260928 02:41:57.541647 10218 raft_consensus.cc:2454] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d4f2a4945e4359bb6f74d6e43d5bca in term 1.
I20260928 02:41:57.541919 10431 leader_election.cc:302] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 7bc859182ead47a29973b4dd95fafb2e, 94d4f2a4945e4359bb6f74d6e43d5bca; no voters:
I20260928 02:41:57.542114 10579 raft_consensus.cc:2781] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:57.542209 10579 raft_consensus.cc:686] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 LEADER]: Becoming Leader. State: Replica: 94d4f2a4945e4359bb6f74d6e43d5bca, State: Running, Role: LEADER
I20260928 02:41:57.542328 10579 consensus_queue.cc:237] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.543004 10602 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: TxnStatusTablet state changed. Reason: New leader 94d4f2a4945e4359bb6f74d6e43d5bca. Latest consensus state: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.543092 10602 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:57.543341 10615 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:41:57.543471 10615 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:41:57.543916 10057 catalog_manager.cc:5495] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca reported cstate change: term changed from 0 to 1, leader changed from <none> to 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3). New cstate: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } health_report { overall_health: HEALTHY } } }
I20260928 02:41:57.555888 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.551785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35845 (local address 127.0.0.1:56368)
0928 02:41:57.551942 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.551958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.552070 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.552376 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.552379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.552390 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:57.552501 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.552508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.553304 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.553307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.554308 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.554320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.554783 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.554787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.554885 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.554937 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.555570 (+ 633us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.555584 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.555653 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.555744 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:41:57.555926 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.551881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.2.123.3:35845)
0928 02:41:57.552069 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.552072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.552085 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:41:57.552192 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.552195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.552258 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.552346 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:57.552593 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.553202 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.554417 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.554679 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.554761 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.555070 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.555466 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.555567 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.555712 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.555766 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
I20260928 02:41:57.559366 10218 raft_consensus.cc:1261] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Refusing update from remote peer 94d4f2a4945e4359bb6f74d6e43d5bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.559531 10358 raft_consensus.cc:1261] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Refusing update from remote peer 94d4f2a4945e4359bb6f74d6e43d5bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.559856 10602 consensus_queue.cc:1035] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.560020 10579 consensus_queue.cc:1035] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.561393 10580 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: TxnStatusTablet state changed. Reason: New leader 94d4f2a4945e4359bb6f74d6e43d5bca. Latest consensus state: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.561499 10580 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:57.561548 10582 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: TxnStatusTablet state changed. Reason: New leader 94d4f2a4945e4359bb6f74d6e43d5bca. Latest consensus state: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.561651 10582 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:57.563401 10602 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.563503 10602 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:57.563596 10587 mvcc.cc:205] Tried to move back new op lower bound from 7334147348718526464 to 7334147348655144960. Current Snapshot: MvccSnapshot[applied={T|T < 7334147348718526464}]
I20260928 02:41:57.564275 10580 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.564379 10580 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 12c3a65b702a4ad0b62223c73bcf6c3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:57.567065 10579 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.567171 10579 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 94d4f2a4945e4359bb6f74d6e43d5bca: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:41:57.567914 10582 tablet_replica.cc:439] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } } }
I20260928 02:41:57.568034 10582 tablet_replica.cc:442] T 2b7ae49066ab48e49e3045b99c861bcc P 7bc859182ead47a29973b4dd95fafb2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:41:57.571275 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.567737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.2.123.3:35845)
0928 02:41:57.568363 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.568368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.568380 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:41:57.568417 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.568420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.568462 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.568819 (+ 357us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:57.568903 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.569480 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.570367 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.570589 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.570685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.570893 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.570961 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.571023 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.571110 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.571158 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542}
I20260928 02:41:57.581291 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.577714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:43705 (local address 127.2.123.3:35845)
0928 02:41:57.577831 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.577836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.578011 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:41:57.578142 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.578146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.578192 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.578267 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:57.578558 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.579327 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.580264 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.580508 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.580611 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.580864 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.580938 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.581002 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.581097 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.581138 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:41:57.581305 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.577641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35845 (local address 127.2.123.3:43705)
0928 02:41:57.577931 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.577942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.578053 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.578316 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.578319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.578326 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:57.578459 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.578465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.579458 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.579461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.580143 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.580153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.580610 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.580613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.580696 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.580737 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.581011 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.581017 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.581084 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.581158 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
I20260928 02:41:57.587859 10358 raft_consensus.cc:1261] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Refusing update from remote peer 94d4f2a4945e4359bb6f74d6e43d5bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.587929 10218 raft_consensus.cc:1261] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Refusing update from remote peer 94d4f2a4945e4359bb6f74d6e43d5bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.588315 10579 consensus_queue.cc:1035] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.588465 10602 consensus_queue.cc:1035] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.624797 10057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:57.624946 10057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:57.626981 10057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: Sending DeleteTablet for 3 replicas of tablet 8688726fe41f49349f995bb08623997f
I20260928 02:41:57.627470 10198 tablet_service.cc:1514] Processing DeleteTablet for tablet 8688726fe41f49349f995bb08623997f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:48282
I20260928 02:41:57.627594 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet 8688726fe41f49349f995bb08623997f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56262
I20260928 02:41:57.628094 10477 tablet_service.cc:1514] Processing DeleteTablet for tablet 8688726fe41f49349f995bb08623997f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56366
I20260928 02:41:57.628379 10629 tablet_replica.cc:330] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: stopping tablet replica
I20260928 02:41:57.628489 10629 raft_consensus.cc:2229] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:57.628597 10629 raft_consensus.cc:2258] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.628796 10630 tablet_replica.cc:330] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: stopping tablet replica
I20260928 02:41:57.628883 10630 raft_consensus.cc:2229] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:57.629161 10630 raft_consensus.cc:2258] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.629802 10629 ts_tablet_manager.cc:1905] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.632769 10629 ts_tablet_manager.cc:1918] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:57.632854 10629 log.cc:1196] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/8688726fe41f49349f995bb08623997f
I20260928 02:41:57.633092 10629 ts_tablet_manager.cc:1939] T 8688726fe41f49349f995bb08623997f P 7bc859182ead47a29973b4dd95fafb2e: Deleting consensus metadata
I20260928 02:41:57.633738 10048 catalog_manager.cc:4841] TS 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715): tablet 8688726fe41f49349f995bb08623997f (table test-table [id=1b4bcce992ee44ab83fcf59d8475f768]) successfully deleted
I20260928 02:41:57.634115 10628 tablet_replica.cc:330] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: stopping tablet replica
I20260928 02:41:57.634152 10630 ts_tablet_manager.cc:1905] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.634334 10628 raft_consensus.cc:2229] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:57.634496 10628 raft_consensus.cc:2258] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.635171 10628 ts_tablet_manager.cc:1905] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.636834 10630 ts_tablet_manager.cc:1918] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:57.636919 10630 log.cc:1196] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/8688726fe41f49349f995bb08623997f
I20260928 02:41:57.637182 10630 ts_tablet_manager.cc:1939] T 8688726fe41f49349f995bb08623997f P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting consensus metadata
I20260928 02:41:57.638094 10628 ts_tablet_manager.cc:1918] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:57.638230 10628 log.cc:1196] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/8688726fe41f49349f995bb08623997f
I20260928 02:41:57.638190 10045 catalog_manager.cc:4841] TS 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845): tablet 8688726fe41f49349f995bb08623997f (table test-table [id=1b4bcce992ee44ab83fcf59d8475f768]) successfully deleted
I20260928 02:41:57.638520 10628 ts_tablet_manager.cc:1939] T 8688726fe41f49349f995bb08623997f P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting consensus metadata
I20260928 02:41:57.639257 10047 catalog_manager.cc:4841] TS 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007): tablet 8688726fe41f49349f995bb08623997f (table test-table [id=1b4bcce992ee44ab83fcf59d8475f768]) successfully deleted
Sep 28 02:41:57 dist-test-slave-rqvb krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:41:57 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:41:57 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:41:57.657564 10610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:41:57.650659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.2.123.62:35411)
0928 02:41:57.650803 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.650808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.650970 (+ 162us) server_negotiation.cc:407] Connection header received
0928 02:41:57.651090 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.651093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.651137 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.651208 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:57.652053 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.652586 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.653794 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.654113 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.654197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.657439 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33110: BlockingRecv error: recv got EOF from 127.0.0.1:33110 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:41:57 dist-test-slave-rqvb krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM
I20260928 02:41:57.667605 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.659242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.2.123.62:35411)
0928 02:41:57.659352 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.659362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.659438 (+ 76us) server_negotiation.cc:407] Connection header received
0928 02:41:57.659561 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.659563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.659599 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.659661 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:41:57.660315 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.660813 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.661895 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.662161 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.662232 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.664636 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:41:57.664640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:41:57.664643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:41:57.664667 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:41:57.666041 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:57.666490 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:57.666493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:57.666494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:57.666576 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:41:57.666855 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:41:57.666858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:41:57.666859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:41:57.667050 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:41:57.667151 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.667369 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.667467 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:41:57.670257 10057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:41:57.670723 10057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:57.675385 10198 tablet_service.cc:1467] Processing CreateTablet for tablet 687df0736c494f789e341169b471ff93 (DEFAULT_TABLE table=test-table [id=b6aceb481ff241f48b5a46cce42446af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.675418 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 687df0736c494f789e341169b471ff93 (DEFAULT_TABLE table=test-table [id=b6aceb481ff241f48b5a46cce42446af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.675693 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687df0736c494f789e341169b471ff93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.675697 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687df0736c494f789e341169b471ff93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.677565 10572 tablet_bootstrap.cc:492] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: Bootstrap starting.
I20260928 02:41:57.678043 10574 tablet_bootstrap.cc:492] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: Bootstrap starting.
I20260928 02:41:57.678371 10572 tablet_bootstrap.cc:654] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.678455 10477 tablet_service.cc:1467] Processing CreateTablet for tablet 687df0736c494f789e341169b471ff93 (DEFAULT_TABLE table=test-table [id=b6aceb481ff241f48b5a46cce42446af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.678730 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687df0736c494f789e341169b471ff93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.678885 10574 tablet_bootstrap.cc:654] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.679906 10572 tablet_bootstrap.cc:492] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: No bootstrap required, opened a new log
I20260928 02:41:57.679916 10574 tablet_bootstrap.cc:492] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: No bootstrap required, opened a new log
I20260928 02:41:57.679998 10574 ts_tablet_manager.cc:1397] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:41:57.679998 10572 ts_tablet_manager.cc:1397] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:41:57.680368 10572 raft_consensus.cc:348] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.680368 10574 raft_consensus.cc:348] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.680471 10572 raft_consensus.cc:374] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.680500 10574 raft_consensus.cc:374] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.680502 10572 raft_consensus.cc:729] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc859182ead47a29973b4dd95fafb2e, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.680531 10574 raft_consensus.cc:729] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c3a65b702a4ad0b62223c73bcf6c3a, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.680625 10572 consensus_queue.cc:260] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.680629 10574 consensus_queue.cc:260] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.680866 10574 ts_tablet_manager.cc:1428] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:57.680866 10572 ts_tablet_manager.cc:1428] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:57.681151 10573 tablet_bootstrap.cc:492] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: Bootstrap starting.
I20260928 02:41:57.682058 10573 tablet_bootstrap.cc:654] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.683897 10573 tablet_bootstrap.cc:492] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: No bootstrap required, opened a new log
I20260928 02:41:57.683980 10573 ts_tablet_manager.cc:1397] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 02:41:57.684696 10573 raft_consensus.cc:348] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.684814 10573 raft_consensus.cc:374] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.684854 10573 raft_consensus.cc:729] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4f2a4945e4359bb6f74d6e43d5bca, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.684971 10573 consensus_queue.cc:260] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.685212 10573 ts_tablet_manager.cc:1428] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:41:57.754534 10580 raft_consensus.cc:482] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:57.754722 10580 raft_consensus.cc:504] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.755828 10580 leader_election.cc:288] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715), 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845)
I20260928 02:41:57.761008 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.756230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:50187 (local address 127.2.123.3:35845)
0928 02:41:57.756399 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.756404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.756687 (+ 283us) server_negotiation.cc:407] Connection header received
0928 02:41:57.756893 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.756897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.756948 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.757022 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:57.757352 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.758121 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.759713 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.760330 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.760443 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.760508 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.760763 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.760866 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:41:57.761201 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.756315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:51797 (local address 127.2.123.2:33715)
0928 02:41:57.756443 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.756447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.756995 (+ 548us) server_negotiation.cc:407] Connection header received
0928 02:41:57.757141 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.757145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.757194 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.757239 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:41:57.757999 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.758744 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.760068 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.760521 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.760627 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.760689 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.760999 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.761084 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:41:57.761372 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.756229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35845 (local address 127.2.123.1:50187)
0928 02:41:57.756570 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.756590 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.756719 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.757067 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.757070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.757089 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:57.757212 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.757217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.758247 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.758250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.759591 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.759603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.760481 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.760485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.760606 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.760635 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.761149 (+ 514us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.761224 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":107,"threads_started":1}
I20260928 02:41:57.761888 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.756549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33715 (local address 127.2.123.1:51797)
0928 02:41:57.756894 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.756906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.757011 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.757370 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.757372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.757387 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:41:57.757817 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.757825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.758886 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.758890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.759951 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.759959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.760746 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.760750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.760857 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.760880 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.761666 (+ 786us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.761739 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260928 02:41:57.761907 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687df0736c494f789e341169b471ff93" candidate_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" is_pre_election: true
I20260928 02:41:57.762077 10498 raft_consensus.cc:2454] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c3a65b702a4ad0b62223c73bcf6c3a in term 0.
I20260928 02:41:57.762326 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687df0736c494f789e341169b471ff93" candidate_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc859182ead47a29973b4dd95fafb2e" is_pre_election: true
I20260928 02:41:57.762429 10149 leader_election.cc:302] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 94d4f2a4945e4359bb6f74d6e43d5bca; no voters:
I20260928 02:41:57.762475 10357 raft_consensus.cc:2454] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c3a65b702a4ad0b62223c73bcf6c3a in term 0.
I20260928 02:41:57.762684 10580 raft_consensus.cc:2781] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:57.762745 10580 raft_consensus.cc:482] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:57.762789 10580 raft_consensus.cc:3037] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.763780 10580 raft_consensus.cc:504] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.764132 10580 leader_election.cc:288] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [CANDIDATE]: Term 1 election: Requested vote from peers 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715), 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845)
I20260928 02:41:57.764401 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687df0736c494f789e341169b471ff93" candidate_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc859182ead47a29973b4dd95fafb2e"
I20260928 02:41:57.764453 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687df0736c494f789e341169b471ff93" candidate_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca"
I20260928 02:41:57.764542 10357 raft_consensus.cc:3037] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.764564 10498 raft_consensus.cc:3037] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.765445 10498 raft_consensus.cc:2454] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c3a65b702a4ad0b62223c73bcf6c3a in term 1.
I20260928 02:41:57.765647 10357 raft_consensus.cc:2454] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c3a65b702a4ad0b62223c73bcf6c3a in term 1.
I20260928 02:41:57.765738 10149 leader_election.cc:302] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 94d4f2a4945e4359bb6f74d6e43d5bca; no voters:
I20260928 02:41:57.765934 10580 raft_consensus.cc:2781] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:57.766176 10580 raft_consensus.cc:686] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 LEADER]: Becoming Leader. State: Replica: 12c3a65b702a4ad0b62223c73bcf6c3a, State: Running, Role: LEADER
I20260928 02:41:57.766431 10580 consensus_queue.cc:237] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.768899 10060 catalog_manager.cc:5495] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1). New cstate: current_term: 1 leader_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:57.788163 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.784551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.2.123.1:42007)
0928 02:41:57.784659 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.784662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.784677 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:41:57.784756 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.784758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.784810 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.784894 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:57.785108 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.785586 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.786777 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.786995 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.787070 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.787323 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.787771 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.787853 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.787936 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.788028 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:41:57.790786 10357 raft_consensus.cc:1261] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Refusing update from remote peer 12c3a65b702a4ad0b62223c73bcf6c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.790855 10498 raft_consensus.cc:1261] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Refusing update from remote peer 12c3a65b702a4ad0b62223c73bcf6c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.791301 10648 consensus_queue.cc:1035] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.791481 10580 consensus_queue.cc:1035] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.806041 10060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:57.806203 10060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:57.808404 10060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: Sending DeleteTablet for 3 replicas of tablet 687df0736c494f789e341169b471ff93
I20260928 02:41:57.808969 10198 tablet_service.cc:1514] Processing DeleteTablet for tablet 687df0736c494f789e341169b471ff93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:48282
I20260928 02:41:57.809145 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet 687df0736c494f789e341169b471ff93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56262
I20260928 02:41:57.809314 10628 tablet_replica.cc:330] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: stopping tablet replica
I20260928 02:41:57.809424 10628 raft_consensus.cc:2229] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:57.809696 10628 raft_consensus.cc:2258] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.809875 10629 tablet_replica.cc:330] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: stopping tablet replica
I20260928 02:41:57.810010 10629 raft_consensus.cc:2229] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:57.810132 10629 raft_consensus.cc:2258] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.810804 10628 ts_tablet_manager.cc:1905] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.810881 10629 ts_tablet_manager.cc:1905] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.813300 10628 ts_tablet_manager.cc:1918] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:57.813378 10628 log.cc:1196] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/687df0736c494f789e341169b471ff93
I20260928 02:41:57.813596 10629 ts_tablet_manager.cc:1918] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:57.813696 10629 log.cc:1196] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/687df0736c494f789e341169b471ff93
I20260928 02:41:57.813697 10628 ts_tablet_manager.cc:1939] T 687df0736c494f789e341169b471ff93 P 12c3a65b702a4ad0b62223c73bcf6c3a: Deleting consensus metadata
I20260928 02:41:57.813937 10629 ts_tablet_manager.cc:1939] T 687df0736c494f789e341169b471ff93 P 7bc859182ead47a29973b4dd95fafb2e: Deleting consensus metadata
I20260928 02:41:57.814472 10047 catalog_manager.cc:4841] TS 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007): tablet 687df0736c494f789e341169b471ff93 (table test-table [id=b6aceb481ff241f48b5a46cce42446af]) successfully deleted
I20260928 02:41:57.814728 10048 catalog_manager.cc:4841] TS 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715): tablet 687df0736c494f789e341169b471ff93 (table test-table [id=b6aceb481ff241f48b5a46cce42446af]) successfully deleted
I20260928 02:41:57.815140 10477 tablet_service.cc:1514] Processing DeleteTablet for tablet 687df0736c494f789e341169b471ff93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56366
I20260928 02:41:57.815315 10630 tablet_replica.cc:330] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: stopping tablet replica
I20260928 02:41:57.815429 10630 raft_consensus.cc:2229] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:57.815550 10630 raft_consensus.cc:2258] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.816103 10060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:41:57.816282 10630 ts_tablet_manager.cc:1905] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:41:57.816530 10060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:41:57.818660 10630 ts_tablet_manager.cc:1918] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:41:57.818723 10630 log.cc:1196] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/687df0736c494f789e341169b471ff93
I20260928 02:41:57.818936 10630 ts_tablet_manager.cc:1939] T 687df0736c494f789e341169b471ff93 P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting consensus metadata
I20260928 02:41:57.819643 10045 catalog_manager.cc:4841] TS 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845): tablet 687df0736c494f789e341169b471ff93 (table test-table [id=b6aceb481ff241f48b5a46cce42446af]) successfully deleted
I20260928 02:41:57.825516 10198 tablet_service.cc:1467] Processing CreateTablet for tablet ba53ccbb1c7c40599711afb289dfbb88 (DEFAULT_TABLE table=test-table [id=9f295054a0f34fbcb1b439b744751706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.825774 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba53ccbb1c7c40599711afb289dfbb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.826303 10338 tablet_service.cc:1467] Processing CreateTablet for tablet ba53ccbb1c7c40599711afb289dfbb88 (DEFAULT_TABLE table=test-table [id=9f295054a0f34fbcb1b439b744751706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.826572 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba53ccbb1c7c40599711afb289dfbb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.828076 10574 tablet_bootstrap.cc:492] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a: Bootstrap starting.
I20260928 02:41:57.828794 10572 tablet_bootstrap.cc:492] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: Bootstrap starting.
I20260928 02:41:57.828999 10574 tablet_bootstrap.cc:654] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.829530 10572 tablet_bootstrap.cc:654] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.830490 10574 tablet_bootstrap.cc:492] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a: No bootstrap required, opened a new log
I20260928 02:41:57.830520 10477 tablet_service.cc:1467] Processing CreateTablet for tablet ba53ccbb1c7c40599711afb289dfbb88 (DEFAULT_TABLE table=test-table [id=9f295054a0f34fbcb1b439b744751706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:41:57.830596 10574 ts_tablet_manager.cc:1397] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:41:57.830785 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba53ccbb1c7c40599711afb289dfbb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:41:57.831058 10574 raft_consensus.cc:348] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.831187 10574 raft_consensus.cc:374] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.831219 10574 raft_consensus.cc:729] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c3a65b702a4ad0b62223c73bcf6c3a, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.831341 10574 consensus_queue.cc:260] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.831611 10574 ts_tablet_manager.cc:1428] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 02:41:57.833000 10573 tablet_bootstrap.cc:492] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: Bootstrap starting.
I20260928 02:41:57.833786 10573 tablet_bootstrap.cc:654] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: Neither blocks nor log segments found. Creating new log.
I20260928 02:41:57.834148 10572 tablet_bootstrap.cc:492] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: No bootstrap required, opened a new log
I20260928 02:41:57.834231 10572 ts_tablet_manager.cc:1397] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 02:41:57.834661 10572 raft_consensus.cc:348] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.834781 10572 raft_consensus.cc:374] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.834815 10572 raft_consensus.cc:729] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc859182ead47a29973b4dd95fafb2e, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.834940 10572 consensus_queue.cc:260] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.835207 10572 ts_tablet_manager.cc:1428] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:41:57.837921 10573 tablet_bootstrap.cc:492] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: No bootstrap required, opened a new log
I20260928 02:41:57.838081 10573 ts_tablet_manager.cc:1397] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 02:41:57.838464 10573 raft_consensus.cc:348] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.838575 10573 raft_consensus.cc:374] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:41:57.838606 10573 raft_consensus.cc:729] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d4f2a4945e4359bb6f74d6e43d5bca, State: Initialized, Role: FOLLOWER
I20260928 02:41:57.838723 10573 consensus_queue.cc:260] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.838990 10573 ts_tablet_manager.cc:1428] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:41:57.906220 10582 raft_consensus.cc:482] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:41:57.906371 10582 raft_consensus.cc:504] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.906804 10582 leader_election.cc:288] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007), 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845)
I20260928 02:41:57.907250 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba53ccbb1c7c40599711afb289dfbb88" candidate_uuid: "7bc859182ead47a29973b4dd95fafb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" is_pre_election: true
I20260928 02:41:57.907282 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba53ccbb1c7c40599711afb289dfbb88" candidate_uuid: "7bc859182ead47a29973b4dd95fafb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" is_pre_election: true
I20260928 02:41:57.907400 10218 raft_consensus.cc:2454] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc859182ead47a29973b4dd95fafb2e in term 0.
I20260928 02:41:57.907456 10498 raft_consensus.cc:2454] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc859182ead47a29973b4dd95fafb2e in term 0.
I20260928 02:41:57.907757 10291 leader_election.cc:302] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 7bc859182ead47a29973b4dd95fafb2e; no voters:
I20260928 02:41:57.908018 10582 raft_consensus.cc:2781] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:41:57.908072 10582 raft_consensus.cc:482] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:41:57.908097 10582 raft_consensus.cc:3037] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.909046 10582 raft_consensus.cc:504] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.909371 10582 leader_election.cc:288] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [CANDIDATE]: Term 1 election: Requested vote from peers 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007), 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845)
I20260928 02:41:57.909674 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba53ccbb1c7c40599711afb289dfbb88" candidate_uuid: "7bc859182ead47a29973b4dd95fafb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a"
I20260928 02:41:57.909705 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba53ccbb1c7c40599711afb289dfbb88" candidate_uuid: "7bc859182ead47a29973b4dd95fafb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca"
I20260928 02:41:57.909780 10218 raft_consensus.cc:3037] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.909811 10498 raft_consensus.cc:3037] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:41:57.910650 10218 raft_consensus.cc:2454] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bc859182ead47a29973b4dd95fafb2e in term 1.
I20260928 02:41:57.910899 10291 leader_election.cc:302] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c3a65b702a4ad0b62223c73bcf6c3a, 7bc859182ead47a29973b4dd95fafb2e; no voters:
I20260928 02:41:57.910957 10498 raft_consensus.cc:2454] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bc859182ead47a29973b4dd95fafb2e in term 1.
I20260928 02:41:57.911067 10582 raft_consensus.cc:2781] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:41:57.911203 10582 raft_consensus.cc:686] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 1 LEADER]: Becoming Leader. State: Replica: 7bc859182ead47a29973b4dd95fafb2e, State: Running, Role: LEADER
I20260928 02:41:57.911346 10582 consensus_queue.cc:237] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } }
I20260928 02:41:57.912889 10057 catalog_manager.cc:5495] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2). New cstate: current_term: 1 leader_uuid: "7bc859182ead47a29973b4dd95fafb2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bc859182ead47a29973b4dd95fafb2e" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 33715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 } health_report { overall_health: UNKNOWN } } }
I20260928 02:41:57.936347 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.933091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.2.123.2:33715)
0928 02:41:57.933199 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.933203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.933214 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:41:57.933275 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.933277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.933315 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.933375 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:57.933630 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.934205 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.935340 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.935571 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.935656 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.935934 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.936002 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.936053 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.936176 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.936222 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:41:57.946750 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.943475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35845 (local address 127.2.123.2:59485)
0928 02:41:57.943797 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.943809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.943916 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.944141 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.944144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.944151 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:57.944250 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.944255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.944932 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.944935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.945600 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.945609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.946076 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.946079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.946164 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.946205 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.946478 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.946484 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.946540 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.946619 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":108,"threads_started":1}
I20260928 02:41:57.946748 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.943574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:59485 (local address 127.2.123.3:35845)
0928 02:41:57.943690 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.943694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.943877 (+ 183us) server_negotiation.cc:407] Connection header received
0928 02:41:57.944005 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.944009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.944061 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.944120 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:57.944346 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.944830 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.945706 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.945922 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.946062 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.946334 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.946404 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.946462 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.946550 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.946600 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:41:57.953639 10218 raft_consensus.cc:1261] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Refusing update from remote peer 7bc859182ead47a29973b4dd95fafb2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.953720 10498 raft_consensus.cc:1261] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Refusing update from remote peer 7bc859182ead47a29973b4dd95fafb2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:41:57.954128 10582 consensus_queue.cc:1035] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d4f2a4945e4359bb6f74d6e43d5bca" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.954301 10658 consensus_queue.cc:1035] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c3a65b702a4ad0b62223c73bcf6c3a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 42007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:41:57.973506 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.969744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:33715 (local address 127.2.123.3:32929)
0928 02:41:57.969857 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:57.969872 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:41:57.969947 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:41:57.970310 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:41:57.970313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:41:57.970320 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:41:57.970411 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.970417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.971358 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.971361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.971984 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:41:57.971995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.972561 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.972565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:41:57.972654 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.972691 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.973186 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.973192 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:41:57.973261 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:41:57.973351 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:41:57.973991 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:57.969933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:32929 (local address 127.2.123.2:33715)
0928 02:41:57.970049 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:41:57.970053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:41:57.970064 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:41:57.970101 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:41:57.970103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:41:57.970144 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:41:57.970199 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:41:57.970512 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.971255 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.972115 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:41:57.972451 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:41:57.972539 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:41:57.973009 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:41:57.973099 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:41:57.973182 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:41:57.973787 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 02:41:57.973846 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:41:57.992450 10057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:57.992600 10057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:41:57.994685 10057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35e1c121f6a143cd81758b96cdd9e7b9: Sending DeleteTablet for 3 replicas of tablet ba53ccbb1c7c40599711afb289dfbb88
I20260928 02:41:57.995323 10198 tablet_service.cc:1514] Processing DeleteTablet for tablet ba53ccbb1c7c40599711afb289dfbb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:48282
I20260928 02:41:57.995396 10477 tablet_service.cc:1514] Processing DeleteTablet for tablet ba53ccbb1c7c40599711afb289dfbb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56366
I20260928 02:41:57.995397 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet ba53ccbb1c7c40599711afb289dfbb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:41:57 UTC) from {username='oryx', principal='oryx/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56262
I20260928 02:41:57.995548 10628 tablet_replica.cc:330] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a: stopping tablet replica
I20260928 02:41:57.995558 10629 tablet_replica.cc:330] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: stopping tablet replica
I20260928 02:41:57.995564 10630 tablet_replica.cc:330] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: stopping tablet replica
I20260928 02:41:57.995657 10628 raft_consensus.cc:2229] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:57.995661 10629 raft_consensus.cc:2229] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:41:57.995659 10630 raft_consensus.cc:2229] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:41:57.995761 10628 raft_consensus.cc:2258] T ba53ccbb1c7c40599711afb289dfbb88 P 12c3a65b702a4ad0b62223c73bcf6c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.995770 10630 raft_consensus.cc:2258] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.995834 10629 raft_consensus.cc:2258] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:41:57.996539 10630 ts_tablet_manager.cc:1905] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.996547 10629 ts_tablet_manager.cc:1905] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:41:57.996673 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 10133
I20260928 02:41:57.999018 10630 ts_tablet_manager.cc:1918] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:57.999099 10630 log.cc:1196] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/ba53ccbb1c7c40599711afb289dfbb88
I20260928 02:41:57.999318 10630 ts_tablet_manager.cc:1939] T ba53ccbb1c7c40599711afb289dfbb88 P 94d4f2a4945e4359bb6f74d6e43d5bca: Deleting consensus metadata
I20260928 02:41:57.999650 10629 ts_tablet_manager.cc:1918] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:41:57.999713 10629 log.cc:1196] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/ba53ccbb1c7c40599711afb289dfbb88
I20260928 02:41:57.999951 10629 ts_tablet_manager.cc:1939] T ba53ccbb1c7c40599711afb289dfbb88 P 7bc859182ead47a29973b4dd95fafb2e: Deleting consensus metadata
I20260928 02:41:57.999969 10045 catalog_manager.cc:4841] TS 94d4f2a4945e4359bb6f74d6e43d5bca (127.2.123.3:35845): tablet ba53ccbb1c7c40599711afb289dfbb88 (table test-table [id=9f295054a0f34fbcb1b439b744751706]) successfully deleted
I20260928 02:41:58.000608 10048 catalog_manager.cc:4841] TS 7bc859182ead47a29973b4dd95fafb2e (127.2.123.2:33715): tablet ba53ccbb1c7c40599711afb289dfbb88 (table test-table [id=9f295054a0f34fbcb1b439b744751706]) successfully deleted
W20260928 02:41:58.004406 10047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:42007 (error 108)
I20260928 02:41:58.004621 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 10273
I20260928 02:41:58.006102 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:41:58.005606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:42007 (local address 127.0.0.1:48310)
0928 02:41:58.005874 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:41:58.005987 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20260928 02:41:58.006215 10047 catalog_manager.cc:4584] TS 12c3a65b702a4ad0b62223c73bcf6c3a (127.2.123.1:42007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba53ccbb1c7c40599711afb289dfbb88: Network error: Client connection negotiation failed: client connection to 127.2.123.1:42007: connect: Connection refused (error 111)
I20260928 02:41:58.010740 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 10413
I20260928 02:41:58.016829 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 10029
2026-09-28T02:41:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3155 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:58 dist-test-slave-rqvb krb5kdc[10670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:58 dist-test-slave-rqvb krb5kdc[10670](info): set up 2 sockets
Sep 28 02:41:58 dist-test-slave-rqvb krb5kdc[10670](info): commencing operation
krb5kdc: starting...
W20260928 02:41:58.974457 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:41:59 dist-test-slave-rqvb krb5kdc[10670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:41:59.090915 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:38799
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:41:59.203521 10683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:41:59.203811 10683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:41:59.203851 10683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:41:59.207939 10683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:41:59.208002 10683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:41:59.208021 10683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:41:59.208036 10683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:41:59.208050 10683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:41:59.213485 10683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:38799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:41:59.214627 10683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:41:59.215651 10683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:41:59.223965 10683 server_base.cc:1034] running on GCE node
W20260928 02:41:59.224053 10689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:59.223999 10691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:41:59.224116 10688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:41:59.224728 10683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:41:59.225805 10683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:41:59.226917 10683 hybrid_clock.cc:648] HybridClock initialized: now 1790563319226900 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:41:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1213 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:41:59 dist-test-slave-rqvb krb5kdc[10696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:41:59 dist-test-slave-rqvb krb5kdc[10696](info): set up 2 sockets
Sep 28 02:41:59 dist-test-slave-rqvb krb5kdc[10696](info): commencing operation
krb5kdc: starting...
W20260928 02:42:00.186357 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:42:00 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:42:00.305498 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41405
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:41405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:00.414623 10709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:00.414891 10709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:00.414944 10709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:00.418905 10709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:00.418963 10709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:00.418990 10709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:00.419013 10709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:00.419040 10709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:00.424263 10709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:41405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:00.425279 10709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:00.426291 10709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:00.434727 10715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:00.434705 10717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:00.434705 10714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:00.434768 10709 server_base.cc:1034] running on GCE node
I20260928 02:42:00.435541 10709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:00.436671 10709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:00.437849 10709 hybrid_clock.cc:648] HybridClock initialized: now 1790563320437830 us; error 32 us; skew 500 ppm
Sep 28 02:42:00 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:00.444736 10709 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:42:00.445786 10709 webserver.cc:460] Webserver started at http://127.2.123.62:41813/ using document root <none> and password file <none>
I20260928 02:42:00.446380 10709 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:00.446450 10709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:00.446674 10709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:00.448714 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "d1a1dec5a68548adb838011362762c27"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "f347fb2d59fac93316c60fa81ccd4e3a"
server_key_iv: "375983b5e531ec191fa762679abca1f1"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.449246 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "d1a1dec5a68548adb838011362762c27"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "f347fb2d59fac93316c60fa81ccd4e3a"
server_key_iv: "375983b5e531ec191fa762679abca1f1"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.453001 10709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:42:00.455444 10724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:00.456657 10709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:42:00.456772 10709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "d1a1dec5a68548adb838011362762c27"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "f347fb2d59fac93316c60fa81ccd4e3a"
server_key_iv: "375983b5e531ec191fa762679abca1f1"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.456866 10709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:00.474793 10709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:00.477741 10709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:00.477918 10709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:00.484659 10709 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:41405
I20260928 02:42:00.484709 10776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:41405 every 8 connection(s)
I20260928 02:42:00.485605 10709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:00.488368 10777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:00.491888 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 10709
I20260928 02:42:00.491998 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:42:00.492265 2540 external_mini_cluster.cc:1428] Setting key d96dd10773d0e3193cec258236e76410
I20260928 02:42:00.494282 10777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Bootstrap starting.
I20260928 02:42:00.497009 10777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:00.497871 10777 log.cc:824] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:00.500290 10777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: No bootstrap required, opened a new log
Sep 28 02:42:00 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:42:00.503963 10777 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } }
I20260928 02:42:00.504225 10777 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:00.504284 10777 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a1dec5a68548adb838011362762c27, State: Initialized, Role: FOLLOWER
I20260928 02:42:00.504777 10777 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } }
I20260928 02:42:00.504933 10777 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:00.505007 10777 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:00.505102 10777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:00.506217 10777 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } }
I20260928 02:42:00.506587 10777 leader_election.cc:302] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a1dec5a68548adb838011362762c27; no voters:
I20260928 02:42:00.506916 10777 leader_election.cc:288] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:00.506996 10782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:00.507165 10782 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 1 LEADER]: Becoming Leader. State: Replica: d1a1dec5a68548adb838011362762c27, State: Running, Role: LEADER
I20260928 02:42:00.507532 10782 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } }
I20260928 02:42:00.508210 10777 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:00.508351 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:00.493781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.2.123.62:41405)
0928 02:42:00.494221 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:00.494240 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:00.494282 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:42:00.495085 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:00.495107 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:00.495527 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:00.495892 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:00.496520 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.498012 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.499144 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.499809 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.499961 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:00.502534 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:00.502541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:00.502558 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:00.502591 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:00.505052 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:00.505477 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:00.505483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:00.505489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:00.505623 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:00.505863 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:00.505866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:00.505868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:00.506417 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:00.506609 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:00.507219 (+ 610us) server_negotiation.cc:299] Negotiation successful
0928 02:42:00.507537 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":147,"threads_started":1}
I20260928 02:42:00.509145 10784 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a1dec5a68548adb838011362762c27. Latest consensus state: current_term: 1 leader_uuid: "d1a1dec5a68548adb838011362762c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } } }
I20260928 02:42:00.509254 10784 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:00.509537 10783 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a1dec5a68548adb838011362762c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } } }
I20260928 02:42:00.509615 10783 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: This master's current role is: LEADER
W20260928 02:42:00.513082 10797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:42:00.513159 10797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:42:00.513218 10791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:00.513849 10791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:00.518860 10791 catalog_manager.cc:1347] Generated new cluster ID: bf7fdf3a50194fa992a46285845dc309
I20260928 02:42:00.518915 10791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:00.540028 10791 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:00.541208 10791 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:00.544811 10791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Generated new TSK 0
I20260928 02:42:00.545459 10791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:42:00.574489 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41405
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:00.685564 10803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:00.685833 10803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:00.685877 10803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:00.689891 10803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:00.689977 10803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:00.690049 10803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:42:00.695384 10803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:00.696458 10803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:00.697467 10803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:00.706715 10809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:00.706720 10808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:00.706849 10803 server_base.cc:1034] running on GCE node
W20260928 02:42:00.706719 10811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:00.707417 10803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:00.708051 10803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:00.709225 10803 hybrid_clock.cc:648] HybridClock initialized: now 1790563320709191 us; error 45 us; skew 500 ppm
Sep 28 02:42:00 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:00.712212 10803 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:42:00.713129 10803 webserver.cc:460] Webserver started at http://127.2.123.1:33719/ using document root <none> and password file <none>
I20260928 02:42:00.713678 10803 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:00.713737 10803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:00.713914 10803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:00.715921 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "cdf5a1f78b1a41e384ef8ab667165e60"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "66f794cff55c7970311fed0c95ce97c5"
server_key_iv: "ba6c8e27ec2e91154d249267ff4af1d9"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.716455 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "cdf5a1f78b1a41e384ef8ab667165e60"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "66f794cff55c7970311fed0c95ce97c5"
server_key_iv: "ba6c8e27ec2e91154d249267ff4af1d9"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.720173 10803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:42:00.722905 10818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:00.724225 10803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:00.724324 10803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "cdf5a1f78b1a41e384ef8ab667165e60"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "66f794cff55c7970311fed0c95ce97c5"
server_key_iv: "ba6c8e27ec2e91154d249267ff4af1d9"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.724431 10803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:00.735188 10803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:00.738257 10803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:00.738430 10803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:00.738996 10803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:00.739802 10803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:00.739850 10803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:00.739892 10803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:00.739919 10803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:00.748761 10803 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:42073
I20260928 02:42:00.748782 10931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:42073 every 8 connection(s)
I20260928 02:42:00.749703 10803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:00.750844 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 10803
I20260928 02:42:00.750972 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:00.751272 2540 external_mini_cluster.cc:1428] Setting key 4cddbee5df76535a1b35c726bfe4bdef
Sep 28 02:42:00 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:42:00.763863 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:00.751652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:54579 (local address 127.2.123.62:41405)
0928 02:42:00.751799 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:00.751804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:00.752965 (+ 1161us) server_negotiation.cc:407] Connection header received
0928 02:42:00.753830 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:00.753833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:00.753885 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:00.753990 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:00.755138 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.755701 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.757336 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.757561 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.757643 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:00.760425 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:00.760428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:00.760430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:00.760454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:00.761933 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:00.762406 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:00.762409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:00.762411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:00.762489 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:00.762805 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:00.762808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:00.762810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:00.762997 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:00.763079 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:00.763581 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 02:42:00.763700 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:42:00.764678 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:00.752091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:54579)
0928 02:42:00.752835 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:00.752875 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:00.753646 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:00.754145 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:00.754153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:00.754510 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:00.754985 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:00.755003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.755818 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.755822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:00.757210 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:00.757221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.757713 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.757718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:00.757916 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:00.758561 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:00.758581 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:00.760266 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:00.762054 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:00.762060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:00.762076 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:00.762315 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:00.762567 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:00.762570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:00.762572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:00.762721 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:00.763088 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:00.763095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:00.763344 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 02:42:00.763525 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 02:42:00.763839 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":120,"threads_started":1}
I20260928 02:42:00.765780 10932 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41405
I20260928 02:42:00.766076 10932 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:00.766695 10932 heartbeater.cc:511] Master 127.2.123.62:41405 requested a full tablet report, sending...
I20260928 02:42:00.768333 10741 ts_manager.cc:194] Registered new tserver with Master: cdf5a1f78b1a41e384ef8ab667165e60 (127.2.123.1:42073)
I20260928 02:42:00.769456 10741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:54579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:42:00.777830 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41405
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:00.886940 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:00.887243 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:00.887315 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:00.891456 10937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:00.891530 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:00.891613 10937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:42:00.897107 10937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:00.898201 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:00.899227 10937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:00.908564 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:00.908653 10942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:00.908703 10937 server_base.cc:1034] running on GCE node
W20260928 02:42:00.908649 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:00.909186 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:00.909840 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:00.911051 10937 hybrid_clock.cc:648] HybridClock initialized: now 1790563320911022 us; error 40 us; skew 500 ppm
Sep 28 02:42:00 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:00.913782 10937 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:42:00.914672 10937 webserver.cc:460] Webserver started at http://127.2.123.2:42541/ using document root <none> and password file <none>
I20260928 02:42:00.915231 10937 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:00.915298 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:00.915498 10937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:00.917531 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "e3cbd7c3e1554f1a925a0eb9628198fe"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "0aae78c4c65965d00893a1d19c3bd57b"
server_key_iv: "4ce2c5602c1389af7982a11b22791b1a"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.918119 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "e3cbd7c3e1554f1a925a0eb9628198fe"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "0aae78c4c65965d00893a1d19c3bd57b"
server_key_iv: "4ce2c5602c1389af7982a11b22791b1a"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.921782 10937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:42:00.924306 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:00.925496 10937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:00.925613 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "e3cbd7c3e1554f1a925a0eb9628198fe"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "0aae78c4c65965d00893a1d19c3bd57b"
server_key_iv: "4ce2c5602c1389af7982a11b22791b1a"
server_key_version: "encryptionkey@0"
I20260928 02:42:00.925712 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:00.953115 10937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:00.956365 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:00.956553 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:00.957136 10937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:00.957949 10937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:00.958017 10937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:00.958071 10937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:00.958097 10937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:00.966747 10937 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:46811
I20260928 02:42:00.966791 11065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:46811 every 8 connection(s)
I20260928 02:42:00.967700 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:00.974259 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 10937
I20260928 02:42:00.974401 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:00.974694 2540 external_mini_cluster.cc:1428] Setting key 208452eeec734ffa22b98bfbb611ff51
Sep 28 02:42:00 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:42:00.982142 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:00.969522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:43407 (local address 127.2.123.62:41405)
0928 02:42:00.969656 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:00.969660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:00.970716 (+ 1056us) server_negotiation.cc:407] Connection header received
0928 02:42:00.971595 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:00.971599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:00.971654 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:00.971727 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:00.972906 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.973510 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.974964 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.975261 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.975369 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:00.978306 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:00.978310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:00.978312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:00.978343 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:00.979822 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:00.980460 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:00.980463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:00.980464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:00.980542 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:00.980940 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:00.980943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:00.980944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:00.981130 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:00.981203 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:00.981869 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 02:42:00.982004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:42:00.983286 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:00.969869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:43407)
0928 02:42:00.970560 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:00.970614 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:00.971397 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:00.971923 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:00.971930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:00.972290 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:00.972754 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:00.972766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.973632 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.973635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:00.974843 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:00.974854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:00.975510 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:00.975514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:00.975671 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:00.976302 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:00.976323 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:00.978142 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:00.979943 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:00.979949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:00.979965 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:00.980294 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:00.980630 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:00.980633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:00.980635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:00.980822 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:00.981230 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:00.981237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:00.981569 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 02:42:00.981801 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 02:42:00.982190 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":116,"threads_started":1}
I20260928 02:42:00.984489 11066 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41405
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:42:00.984738 11066 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:00.985235 11066 heartbeater.cc:511] Master 127.2.123.62:41405 requested a full tablet report, sending...
I20260928 02:42:00.986352 10741 ts_manager.cc:194] Registered new tserver with Master: e3cbd7c3e1554f1a925a0eb9628198fe (127.2.123.2:46811)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:42:00.987011 10741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:43407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:42:01.001327 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41405
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:01.110636 11071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:01.110957 11071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:01.111018 11071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:01.115216 11071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:01.115281 11071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:01.115371 11071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:42:01.120635 11071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:01.121677 11071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:01.122690 11071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:01.131999 11076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:01.132296 11079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:01.132153 11077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:01.132139 11071 server_base.cc:1034] running on GCE node
I20260928 02:42:01.132952 11071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:01.133661 11071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:01.134845 11071 hybrid_clock.cc:648] HybridClock initialized: now 1790563321134811 us; error 46 us; skew 500 ppm
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:01.137494 11071 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:42:01.138417 11071 webserver.cc:460] Webserver started at http://127.2.123.3:38321/ using document root <none> and password file <none>
I20260928 02:42:01.138943 11071 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:01.139007 11071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:01.139187 11071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:01.141202 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "937a554e8959425ca7163dd5f91da153"
format_stamp: "Formatted at 2026-09-28 02:42:01 on dist-test-slave-rqvb"
server_key: "f4072722465ef70c34dbd2eb0056980b"
server_key_iv: "64fdae7c9cefe6dfe9bb52b90f3a2cc6"
server_key_version: "encryptionkey@0"
I20260928 02:42:01.141741 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "937a554e8959425ca7163dd5f91da153"
format_stamp: "Formatted at 2026-09-28 02:42:01 on dist-test-slave-rqvb"
server_key: "f4072722465ef70c34dbd2eb0056980b"
server_key_iv: "64fdae7c9cefe6dfe9bb52b90f3a2cc6"
server_key_version: "encryptionkey@0"
I20260928 02:42:01.145475 11071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:42:01.148012 11086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.149322 11071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:01.149431 11071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "937a554e8959425ca7163dd5f91da153"
format_stamp: "Formatted at 2026-09-28 02:42:01 on dist-test-slave-rqvb"
server_key: "f4072722465ef70c34dbd2eb0056980b"
server_key_iv: "64fdae7c9cefe6dfe9bb52b90f3a2cc6"
server_key_version: "encryptionkey@0"
I20260928 02:42:01.149525 11071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:01.173040 11071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:01.176327 11071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:01.176513 11071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:01.177109 11071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:01.177995 11071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:01.178042 11071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.178094 11071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:01.178122 11071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.186304 11071 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:36427
I20260928 02:42:01.186340 11199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:36427 every 8 connection(s)
I20260928 02:42:01.187273 11071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:01.187770 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 11071
I20260928 02:42:01.187893 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:01.188164 2540 external_mini_cluster.cc:1428] Setting key de2d0d086c74dd261ef1f8c12a7cb221
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:42:01.200352 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:01.189030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:59183 (local address 127.2.123.62:41405)
0928 02:42:01.189173 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:01.189177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:01.190428 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 02:42:01.191287 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:01.191290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:01.191341 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:01.191414 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:01.192567 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.193110 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.194295 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.194513 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.194598 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.197051 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:01.197054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:01.197056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:01.197082 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:01.198457 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:01.198957 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:01.198959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:01.198960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:01.199029 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:01.199405 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:01.199407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:01.199408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:01.199581 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:01.199658 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:01.200117 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 02:42:01.200220 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:42:01.201246 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:01.189508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:59183)
0928 02:42:01.190289 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:01.190344 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:01.191102 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:01.191583 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:01.191591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:01.191957 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:01.192435 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.192450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.193224 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.193227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.194188 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.194197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.194646 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.194650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.194828 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.195372 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:01.195388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:01.196845 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:01.198574 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:01.198579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:01.198592 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:01.198851 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:01.199130 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:01.199133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:01.199134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:01.199287 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:01.199657 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:01.199662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:01.199908 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 02:42:01.200064 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 02:42:01.200330 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":120,"threads_started":1}
I20260928 02:42:01.202363 11200 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41405
I20260928 02:42:01.202598 11200 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:01.203086 11200 heartbeater.cc:511] Master 127.2.123.62:41405 requested a full tablet report, sending...
I20260928 02:42:01.204118 10739 ts_manager.cc:194] Registered new tserver with Master: 937a554e8959425ca7163dd5f91da153 (127.2.123.3:36427)
I20260928 02:42:01.204808 10739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:59183
I20260928 02:42:01.213996 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab.
Entry for principal oryx/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab.
WARNING: no policy specified for oryx/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab.
Entry for principal oryx/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab.
WARNING: no policy specified for oryx/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab.
Entry for principal oryx/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab.
I20260928 02:42:01.286407 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 10803
I20260928 02:42:01.292785 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 10937
I20260928 02:42:01.297159 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 11071
I20260928 02:42:01.301761 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 10709
I20260928 02:42:01.306979 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41405
--webserver_interface=127.2.123.62
--webserver_port=41813
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:41405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:01.416621 11209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:01.416949 11209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:01.417019 11209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:01.421474 11209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:01.421564 11209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:01.421592 11209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:01.421613 11209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:01.421634 11209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:01.427062 11209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:41405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=41813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:01.428167 11209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:01.429214 11209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:01.437583 11217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:01.437615 11215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:01.437642 11214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:01.437836 11209 server_base.cc:1034] running on GCE node
I20260928 02:42:01.438480 11209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:01.439572 11209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:01.440742 11209 hybrid_clock.cc:648] HybridClock initialized: now 1790563321440717 us; error 39 us; skew 500 ppm
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:01.443470 11209 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:42:01.444331 11209 webserver.cc:460] Webserver started at http://127.2.123.62:41813/ using document root <none> and password file <none>
I20260928 02:42:01.444859 11209 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:01.444939 11209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:01.448794 11209 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:01.450941 11224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.452337 11209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:01.452454 11209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "d1a1dec5a68548adb838011362762c27"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "f347fb2d59fac93316c60fa81ccd4e3a"
server_key_iv: "375983b5e531ec191fa762679abca1f1"
server_key_version: "encryptionkey@0"
I20260928 02:42:01.453004 11209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:01.482254 11209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:01.485333 11209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:01.485518 11209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:01.493135 11209 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:41405
I20260928 02:42:01.493158 11276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:41405 every 8 connection(s)
I20260928 02:42:01.494306 11209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:01.498662 11277 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:42:01.500312 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Bootstrap starting.
I20260928 02:42:01.503341 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 11209
I20260928 02:42:01.503793 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:42073
--local_ip_for_outbound_sockets=127.2.123.1
--tserver_master_addrs=127.2.123.62:41405
--webserver_port=33719
--webserver_interface=127.2.123.1
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.123.1
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:42:01.517285 11277 log.cc:824] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:01.523924 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:42:01.524279 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Bootstrap complete.
I20260928 02:42:01.528460 11277 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } }
I20260928 02:42:01.528867 11277 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a1dec5a68548adb838011362762c27, State: Initialized, Role: FOLLOWER
I20260928 02:42:01.529340 11277 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } }
I20260928 02:42:01.529444 11277 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:01.529505 11277 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:01.529592 11277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:42:01.531486 11277 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } }
I20260928 02:42:01.531821 11277 leader_election.cc:302] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a1dec5a68548adb838011362762c27; no voters:
I20260928 02:42:01.532119 11277 leader_election.cc:288] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:42:01.532207 11281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:42:01.532389 11281 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [term 2 LEADER]: Becoming Leader. State: Replica: d1a1dec5a68548adb838011362762c27, State: Running, Role: LEADER
I20260928 02:42:01.532754 11281 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } }
I20260928 02:42:01.533320 11277 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:01.534176 11282 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d1a1dec5a68548adb838011362762c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } } }
I20260928 02:42:01.534296 11282 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:01.534543 11283 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a1dec5a68548adb838011362762c27. Latest consensus state: current_term: 2 leader_uuid: "d1a1dec5a68548adb838011362762c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a1dec5a68548adb838011362762c27" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41405 } } }
I20260928 02:42:01.534612 11283 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:01.534633 11290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:01.537949 11290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:01.538673 11290 catalog_manager.cc:1259] Loaded cluster ID: bf7fdf3a50194fa992a46285845dc309
I20260928 02:42:01.538741 11290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:01.541067 11290 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:01.541738 11290 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1a1dec5a68548adb838011362762c27: Loaded TSK: 0
I20260928 02:42:01.543108 11290 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:42:01.616060 11279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:01.616348 11279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:01.616410 11279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:01.620632 11279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:01.620715 11279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:01.620800 11279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:42:01.626255 11279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.1.keytab
--principal=oryx/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:42073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=33719
--tserver_master_addrs=127.2.123.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:01.627305 11279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:01.628334 11279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:01.638193 11303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:01.638103 11304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:01.638100 11306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:01.638636 11279 server_base.cc:1034] running on GCE node
I20260928 02:42:01.639065 11279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:01.639729 11279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:01.640930 11279 hybrid_clock.cc:648] HybridClock initialized: now 1790563321640895 us; error 52 us; skew 500 ppm
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563321, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:01.643963 11279 init.cc:377] Logged in from keytab as oryx/127.2.123.1@KRBTEST.COM (short username oryx)
I20260928 02:42:01.644934 11279 webserver.cc:460] Webserver started at http://127.2.123.1:33719/ using document root <none> and password file <none>
I20260928 02:42:01.645474 11279 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:01.645551 11279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:01.649616 11279 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:42:01.651854 11313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.653126 11279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:01.653232 11279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "cdf5a1f78b1a41e384ef8ab667165e60"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "66f794cff55c7970311fed0c95ce97c5"
server_key_iv: "ba6c8e27ec2e91154d249267ff4af1d9"
server_key_version: "encryptionkey@0"
I20260928 02:42:01.653875 11279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:01.670702 11279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:01.673765 11279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:01.673933 11279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:01.674543 11279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:01.675328 11279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:01.675374 11279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.675411 11279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:01.675426 11279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.683710 11279 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:42073
I20260928 02:42:01.683774 11426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:42073 every 8 connection(s)
I20260928 02:42:01.684702 11279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:01.690377 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 11279
I20260928 02:42:01.690860 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:46811
--local_ip_for_outbound_sockets=127.2.123.2
--tserver_master_addrs=127.2.123.62:41405
--webserver_port=42541
--webserver_interface=127.2.123.2
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.123.2
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:01.698105 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:01.686921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:37223)
0928 02:42:01.687679 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:01.687732 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:01.688502 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:01.689622 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:01.689630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:01.690023 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:01.690699 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.690719 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.692130 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.692133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.693010 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.693019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.694121 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.694126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.694269 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.694944 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:01.694971 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:01.697291 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":171,"threads_started":1}
W20260928 02:42:01.698580 11429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:01.686787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:37223 (local address 127.2.123.62:41405)
0928 02:42:01.687276 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:01.687292 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:01.687870 (+ 578us) server_negotiation.cc:407] Connection header received
0928 02:42:01.688876 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:01.688888 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:01.689193 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:01.689475 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:01.690882 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.691982 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.693131 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.693834 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.694023 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.697585 (+ 3562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:37223: BlockingRecv error: recv got EOF from 127.2.123.1:37223 (error 108)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":154,"threads_started":1}
W20260928 02:42:01.698711 11427 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:41405 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.123.62:41405: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:01.704617 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:01.699169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:57555)
0928 02:42:01.699266 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:01.699280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:01.699357 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:01.699640 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:01.699642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:01.699786 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:01.699878 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.699883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.700898 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.700901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.701507 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.701515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.702096 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.702099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.702182 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.702634 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:01.702645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:01.704479 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:42:01.704728 11429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:01.699260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:57555 (local address 127.2.123.62:41405)
0928 02:42:01.699374 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:01.699378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:01.699412 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:42:01.699457 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:01.699459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:01.699509 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:01.699596 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:01.699990 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.700772 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.701623 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.701997 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.702093 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.704619 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:57555: BlockingRecv error: recv got EOF from 127.2.123.1:57555 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:01.709734 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:01.705300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:57101)
0928 02:42:01.705408 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:01.705422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:01.705508 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:01.705721 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:01.705724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:01.705853 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:01.705918 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.705922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.706701 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.706704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.707213 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.707221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.707609 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.707612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.707683 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.708112 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:01.708123 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:01.709612 (+ 1489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 02:42:01.709830 11429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:01.705375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:57101 (local address 127.2.123.62:41405)
0928 02:42:01.705504 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:01.705508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:01.705534 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:42:01.705593 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:01.705596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:01.705636 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:01.705696 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:01.706116 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.706606 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.707311 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.707519 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.707601 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.709736 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:57101: BlockingRecv error: recv got EOF from 127.2.123.1:57101 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 02:42:01.710150 11427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:42:01.801996 11431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:01.802286 11431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:01.802330 11431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:01.806373 11431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:01.806433 11431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:01.806494 11431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:42:01.811717 11431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.2.keytab
--principal=oryx/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:46811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=42541
--tserver_master_addrs=127.2.123.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:01.812737 11431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:01.813719 11431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:01.822867 11436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:01.822995 11437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:01.823060 11431 server_base.cc:1034] running on GCE node
W20260928 02:42:01.822865 11439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:01.823604 11431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:01.824235 11431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:01.825407 11431 hybrid_clock.cc:648] HybridClock initialized: now 1790563321825377 us; error 41 us; skew 500 ppm
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563321, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:01.828167 11431 init.cc:377] Logged in from keytab as oryx/127.2.123.2@KRBTEST.COM (short username oryx)
I20260928 02:42:01.829020 11431 webserver.cc:460] Webserver started at http://127.2.123.2:42541/ using document root <none> and password file <none>
I20260928 02:42:01.829562 11431 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:01.829643 11431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:01.833567 11431 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:01.835855 11446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.837180 11431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:01.837298 11431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "e3cbd7c3e1554f1a925a0eb9628198fe"
format_stamp: "Formatted at 2026-09-28 02:42:00 on dist-test-slave-rqvb"
server_key: "0aae78c4c65965d00893a1d19c3bd57b"
server_key_iv: "4ce2c5602c1389af7982a11b22791b1a"
server_key_version: "encryptionkey@0"
I20260928 02:42:01.837903 11431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:01.852348 11431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:01.855333 11431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:01.855518 11431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:01.856092 11431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:01.856973 11431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:01.857019 11431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.857072 11431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:01.857100 11431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:01.866181 11431 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:46811
I20260928 02:42:01.866216 11559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:46811 every 8 connection(s)
I20260928 02:42:01.867220 11431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:01.867542 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 11431
I20260928 02:42:01.868129 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:36427
--local_ip_for_outbound_sockets=127.2.123.3
--tserver_master_addrs=127.2.123.62:41405
--webserver_port=38321
--webserver_interface=127.2.123.3
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.123.3
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:42:01.881340 11429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:01.869307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:48319 (local address 127.2.123.62:41405)
0928 02:42:01.869467 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:01.869472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:01.870733 (+ 1261us) server_negotiation.cc:407] Connection header received
0928 02:42:01.871780 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:01.871783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:01.871873 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:01.872026 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:01.873601 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.874436 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.876066 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.876369 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.876462 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.881235 (+ 4773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:48319: BlockingRecv error: recv got EOF from 127.2.123.2:48319 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:42:01.881469 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:01.869819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:48319)
0928 02:42:01.870574 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:01.870616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:01.871525 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:01.872305 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:01.872315 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:01.872794 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:01.873435 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.873454 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.874566 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.874569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.875947 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.875958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.876515 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.876519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.876699 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.877303 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:01.877323 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:01.880372 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":170,"threads_started":1}
W20260928 02:42:01.881948 11560 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:41405 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.123.62:41405: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:01.887353 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:01.882451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:35545)
0928 02:42:01.882565 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:01.882579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:01.882663 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:01.882884 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:01.882966 (+ 82us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:01.883107 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:01.883199 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.883205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.883928 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.883931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.884475 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.884483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.884906 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.884910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.884998 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.885426 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:01.885448 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:01.887223 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:42:01.887738 11429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:01.882533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:35545 (local address 127.2.123.62:41405)
0928 02:42:01.882642 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:01.882646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:01.882658 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:42:01.882747 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:01.882750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:01.882793 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:01.882859 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:01.883315 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.883821 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.884582 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.884803 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.884887 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.887406 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:35545: BlockingRecv error: recv got EOF from 127.2.123.2:35545 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:01 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:01.893285 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:01.887852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:50119)
0928 02:42:01.888086 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:01.888099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:01.888169 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:01.888377 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:01.888380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:01.888533 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:01.888632 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.888638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.889358 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.889361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.890229 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:01.890238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.890642 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.890646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:01.890721 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.891178 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:01.891192 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:01.893161 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260928 02:42:01.893498 11429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:01.888065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:50119 (local address 127.2.123.62:41405)
0928 02:42:01.888161 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:01.888164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:01.888176 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:42:01.888246 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:01.888248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:01.888283 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:01.888352 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:01.888756 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.889255 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.890331 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:01.890539 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:01.890623 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:01.893290 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:50119: BlockingRecv error: recv got EOF from 127.2.123.2:50119 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 02:42:01.893567 11560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:42:01.984279 11562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:01.984573 11562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:01.984617 11562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:01.988705 11562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:01.988766 11562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:01.988824 11562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:42:01.994195 11562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/krb5kdc/oryx_127.2.123.3.keytab
--principal=oryx/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:36427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=38321
--tserver_master_addrs=127.2.123.62:41405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:01.995312 11562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:01.996346 11562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:02.005867 11569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:02.005861 11568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:02.005867 11571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:02.006084 11562 server_base.cc:1034] running on GCE node
I20260928 02:42:02.006635 11562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:02.007335 11562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:02.008514 11562 hybrid_clock.cc:648] HybridClock initialized: now 1790563322008490 us; error 36 us; skew 500 ppm
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563322, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:02.011400 11562 init.cc:377] Logged in from keytab as oryx/127.2.123.3@KRBTEST.COM (short username oryx)
I20260928 02:42:02.012359 11562 webserver.cc:460] Webserver started at http://127.2.123.3:38321/ using document root <none> and password file <none>
I20260928 02:42:02.012912 11562 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:02.012993 11562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:02.017098 11562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:42:02.019558 11578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.020843 11562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:42:02.020974 11562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "937a554e8959425ca7163dd5f91da153"
format_stamp: "Formatted at 2026-09-28 02:42:01 on dist-test-slave-rqvb"
server_key: "f4072722465ef70c34dbd2eb0056980b"
server_key_iv: "64fdae7c9cefe6dfe9bb52b90f3a2cc6"
server_key_version: "encryptionkey@0"
I20260928 02:42:02.021552 11562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:02.039891 11562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:02.043094 11562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:02.043272 11562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:02.043852 11562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:02.044678 11562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:02.044723 11562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.044760 11562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:02.044780 11562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:02.053174 11562 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:36427
I20260928 02:42:02.053211 11691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:36427 every 8 connection(s)
I20260928 02:42:02.054343 11562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:02.055405 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 11562
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:02.065732 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.056126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.2.123.62:41405)
0928 02:42:02.056278 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.056284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.056635 (+ 351us) server_negotiation.cc:407] Connection header received
0928 02:42:02.056798 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.056802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.056857 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.056946 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.057778 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.058386 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.059330 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.059558 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.059641 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.061201 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:02.061208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:02.061228 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:02.061264 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:02.063295 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.063810 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.063817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.063824 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.063982 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:02.064367 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:02.064370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:02.064372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:02.064822 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:02.064982 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:02.065313 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 02:42:02.065574 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 02:42:02.066680 11695 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:02.056692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:37923 (local address 127.2.123.62:41405)
0928 02:42:02.056877 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.056882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.057917 (+ 1035us) server_negotiation.cc:407] Connection header received
0928 02:42:02.058975 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.058978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.059026 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.059111 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.060272 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.061051 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.062720 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.062948 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.063025 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.066561 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:37923: BlockingRecv error: recv got EOF from 127.2.123.3:37923 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260928 02:42:02.067245 11696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.056929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:37923)
0928 02:42:02.057737 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.057815 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.058773 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.059260 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.059267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.059629 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:02.060133 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.060148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.061188 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.061191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.062580 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.062592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.063101 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.063106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.063314 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.064006 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:02.064027 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:02.066427 (+ 2400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":132,"threads_started":1}
W20260928 02:42:02.067703 11692 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:41405 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.123.62:41405: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:02.073122 11696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.068109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:58629)
0928 02:42:02.068294 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.068309 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.068382 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.068744 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.068747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.068905 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:02.068998 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.069003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.069710 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.069714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.070320 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.070328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.070731 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.070734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.070820 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.071242 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:02.071253 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:02.072992 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":85}
W20260928 02:42:02.073220 11429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:02.068169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:58629 (local address 127.2.123.62:41405)
0928 02:42:02.068561 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.068565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.068577 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:42:02.068609 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.068610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.068645 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.068709 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.069104 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.069614 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.070431 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.070639 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.070720 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.073124 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:58629: BlockingRecv error: recv got EOF from 127.2.123.3:58629 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:02.078085 11696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.073687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:53337)
0928 02:42:02.073798 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.073812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.073888 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.074116 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.074118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.074238 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:02.074310 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.074334 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.075008 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.075011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.075567 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.075575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.075992 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.075994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.076055 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.076448 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:02.076457 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:02.077981 (+ 1524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 02:42:02.078162 11429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:02.073765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:53337 (local address 127.2.123.62:41405)
0928 02:42:02.073913 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.073917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.073928 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:42:02.073977 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.073980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.074022 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.074086 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.074420 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.074900 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.075661 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.075872 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.075947 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.078060 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:53337: BlockingRecv error: recv got EOF from 127.2.123.3:53337 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 02:42:02.078299 11692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:02.716208 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.710766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:52933)
0928 02:42:02.711036 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.711050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.711149 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.711413 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.711416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.711644 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:02.711747 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.711753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.712518 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.712521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.713159 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.713169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.713661 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.713664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.713730 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.714245 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:02.714256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:02.716084 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
W20260928 02:42:02.716349 11698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:02.710911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:52933 (local address 127.2.123.62:41405)
0928 02:42:02.711123 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.711126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.711139 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:42:02.711232 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.711234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.711277 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.711350 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.711862 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.712402 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.713281 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.713506 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.713640 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.716221 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:52933: BlockingRecv error: recv got EOF from 127.2.123.1:52933 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:02 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:02.899820 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:02.894256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:52957)
0928 02:42:02.894500 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:02.894514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:02.894609 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:02.894832 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:02.894834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:02.895010 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:02.895101 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.895106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.895973 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.895976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.896570 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:02.896579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.897074 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.897078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:02.897149 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.897701 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:02.897715 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:02.899697 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
W20260928 02:42:02.899946 11698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:02.894382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:52957 (local address 127.2.123.62:41405)
0928 02:42:02.894555 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:02.894558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:02.894573 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:42:02.894688 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:02.894690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:02.894733 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:02.894810 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:02.895240 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.895835 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.896695 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:02.896930 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:02.897047 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:02.899839 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:52957: BlockingRecv error: recv got EOF from 127.2.123.2:52957 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 02:42:03 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:03 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:03.084270 11700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:03.078919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:42161)
0928 02:42:03.079174 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:03.079188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:03.079288 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:03.079515 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:03.079517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:03.079681 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:03.079808 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:03.079813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.080640 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.080643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:03.081223 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:03.081231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.081678 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.081680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:03.081751 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:03.082264 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:03.082276 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:03.084135 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
W20260928 02:42:03.084403 11698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:03.079026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:42161 (local address 127.2.123.62:41405)
0928 02:42:03.079173 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:03.079176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:03.079258 (+ 82us) server_negotiation.cc:407] Connection header received
0928 02:42:03.079359 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:03.079361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:03.079410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:03.079486 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:03.079953 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.080506 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.081332 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.081571 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.081649 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:03.084303 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:42161: BlockingRecv error: recv got EOF from 127.2.123.3:42161 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 02:42:03 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:03 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:03.722662 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:03.716995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:53333)
0928 02:42:03.717328 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:03.717346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:03.717472 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:03.717737 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:03.717739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:03.717925 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:03.718079 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:03.718086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.718849 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.718852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:03.719421 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:03.719431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.719923 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.719927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:03.720014 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:03.720550 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:03.720562 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:03.722455 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":132,"threads_started":1}
W20260928 02:42:03.722754 11702 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:03.717157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:53333 (local address 127.2.123.62:41405)
0928 02:42:03.717394 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:03.717398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:03.717413 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:42:03.717574 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:03.717577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:03.717625 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:03.717698 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:03.718190 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.718734 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.719559 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.719797 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.719872 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:03.722630 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:53333: BlockingRecv error: recv got EOF from 127.2.123.1:53333 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
Sep 28 02:42:03 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:03 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:03.905918 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:03.900549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:37375)
0928 02:42:03.900801 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:03.900815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:03.900924 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:03.901220 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:03.901223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:03.901389 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:03.901491 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:03.901496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.902318 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.902321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:03.902870 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:03.902877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.903348 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.903350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:03.903413 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:03.903875 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:03.903887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:03.905779 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
W20260928 02:42:03.906067 11702 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:03.900680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:37375 (local address 127.2.123.62:41405)
0928 02:42:03.900829 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:03.900832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:03.900891 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:42:03.901068 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:03.901070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:03.901116 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:03.901182 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:03.901609 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.902200 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.903035 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:03.903251 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:03.903329 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:03.905925 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:37375: BlockingRecv error: recv got EOF from 127.2.123.2:37375 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 02:42:04 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:04 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:04.091166 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:04.085211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:41395)
0928 02:42:04.085512 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:04.085526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:04.085618 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:04.086028 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:04.086031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:04.086212 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:04.086311 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:04.086316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.087116 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.087119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:04.087789 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:04.087800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.088335 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.088338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:04.088448 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:04.089036 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:04.089049 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:04.091020 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":119,"threads_started":1}
W20260928 02:42:04.091277 11702 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:04.085418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:41395 (local address 127.2.123.62:41405)
0928 02:42:04.085574 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:04.085577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:04.085593 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:42:04.085763 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:04.085767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:04.085824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:04.085909 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:04.086435 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.086980 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.087942 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.088187 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.088324 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:04.091177 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:41395: BlockingRecv error: recv got EOF from 127.2.123.3:41395 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 02:42:04 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:04 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:04.728957 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:04.723470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:42057)
0928 02:42:04.723732 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:04.723747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:04.723853 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:04.724148 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:04.724151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:04.724322 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:04.724418 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:04.724423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.725300 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.725304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:04.726020 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:04.726031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.726445 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.726448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:04.726550 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:04.727011 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:04.727025 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:04.728790 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
W20260928 02:42:04.729079 11706 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:04.723577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:42057 (local address 127.2.123.62:41405)
0928 02:42:04.723828 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:04.723832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:04.723846 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:42:04.723971 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:04.723974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:04.724023 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:04.724106 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:04.724528 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.725191 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.726113 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.726349 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.726424 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:04.728948 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:42057: BlockingRecv error: recv got EOF from 127.2.123.1:42057 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
Sep 28 02:42:04 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:04 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:04.912173 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:04.906734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:58487)
0928 02:42:04.907156 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:04.907172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:04.907283 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:04.907553 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:04.907555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:04.907725 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:04.907820 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:04.907825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.908571 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.908574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:04.909126 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:04.909134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.909574 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.909577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:04.909637 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:04.910206 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:04.910218 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:04.912032 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":80,"threads_started":1}
W20260928 02:42:04.912277 11706 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:04.906807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:58487 (local address 127.2.123.62:41405)
0928 02:42:04.906968 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:04.906973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:04.907246 (+ 273us) server_negotiation.cc:407] Connection header received
0928 02:42:04.907381 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:04.907384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:04.907438 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:04.907523 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:04.907908 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.908428 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.909232 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:04.909447 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:04.909532 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:04.912168 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:58487: BlockingRecv error: recv got EOF from 127.2.123.2:58487 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:42:05 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:05 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:05.098101 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:05.092064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:53081)
0928 02:42:05.092440 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:05.092454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:05.092575 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:05.092846 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:05.092848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:05.093053 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:05.093160 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:05.093166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.093908 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.093911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:05.094580 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:05.094589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.095100 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.095103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:05.095173 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:05.095781 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:05.095793 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:05.097934 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":99,"threads_started":1}
W20260928 02:42:05.098205 11706 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:05.092169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:53081 (local address 127.2.123.62:41405)
0928 02:42:05.092294 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:05.092298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:05.092576 (+ 278us) server_negotiation.cc:407] Connection header received
0928 02:42:05.092672 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:05.092675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:05.092726 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:05.092788 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:05.093264 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.093780 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.094731 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.094979 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.095091 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:05.098107 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:53081: BlockingRecv error: recv got EOF from 127.2.123.3:53081 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:42:05 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:05 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:05.740537 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:05.729802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:33097)
0928 02:42:05.730073 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:05.730090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:05.730196 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:05.730482 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:05.730484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:05.730666 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:05.730765 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:05.730771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.731890 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.731894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:05.732458 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:05.732466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.733065 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.733069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:05.733157 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:05.738115 (+ 4958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:05.738181 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:05.740343 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
W20260928 02:42:05.740687 11710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:05.729903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:33097 (local address 127.2.123.62:41405)
0928 02:42:05.730210 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:05.730215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:05.730232 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:42:05.730284 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:05.730287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:05.730341 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:05.730445 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:05.730875 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.731756 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.732567 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.732934 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.733037 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:05.740538 (+ 7501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:33097: BlockingRecv error: recv got EOF from 127.2.123.1:33097 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
Sep 28 02:42:05 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:05 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:05.918394 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:05.912897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:36697)
0928 02:42:05.913172 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:05.913185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:05.913276 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:05.913503 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:05.913505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:05.913670 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:05.913780 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:05.913785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.914563 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.914566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:05.915193 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:05.915202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.915706 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.915709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:05.915785 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:05.916341 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:05.916354 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:05.918262 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
W20260928 02:42:05.918562 11710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:05.913062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:36697 (local address 127.2.123.62:41405)
0928 02:42:05.913191 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:05.913195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:05.913238 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:42:05.913355 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:05.913357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:05.913400 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:05.913462 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:05.913863 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.914420 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.915360 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:05.915595 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:05.915679 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:05.918426 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:36697: BlockingRecv error: recv got EOF from 127.2.123.2:36697 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 02:42:06 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:06 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:06.104286 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:06.098956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:41809)
0928 02:42:06.099215 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:06.099229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:06.099327 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:06.099600 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:06.099603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:06.099804 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:06.099907 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:06.099914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.100694 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.100697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:06.101278 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:06.101291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.101744 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.101747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:06.101815 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:06.102349 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:06.102360 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:06.104166 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
W20260928 02:42:06.104555 11710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:06.099080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:41809 (local address 127.2.123.62:41405)
0928 02:42:06.099209 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:06.099213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:06.099279 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:42:06.099400 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:06.099403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:06.099446 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:06.099503 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:06.100064 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.100578 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.101407 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.101634 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.101714 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:06.104397 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:41809: BlockingRecv error: recv got EOF from 127.2.123.3:41809 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 02:42:06 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:06 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:06.747331 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:06.741480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:35065)
0928 02:42:06.741867 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:06.741882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:06.742016 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:06.742306 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:06.742308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:06.742513 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:06.742664 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:06.742671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.743415 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.743418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:06.744006 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:06.744014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.744445 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.744448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:06.744527 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:06.745048 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:06.745060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:06.747217 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":184,"threads_started":1}
W20260928 02:42:06.747443 11714 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:06.741590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:35065 (local address 127.2.123.62:41405)
0928 02:42:06.741866 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:06.741871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:06.741987 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:42:06.742145 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:06.742147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:06.742191 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:06.742265 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:06.742784 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.743313 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.744123 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.744345 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.744431 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:06.747335 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:35065: BlockingRecv error: recv got EOF from 127.2.123.1:35065 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
Sep 28 02:42:06 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:06 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:06.924450 11715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:06.919250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:39495)
0928 02:42:06.919512 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:06.919528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:06.919606 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:06.919863 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:06.919866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:06.920076 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:06.920171 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:06.920177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.920881 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.920884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:06.921480 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:06.921488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.922036 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.922040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:06.922123 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:06.922646 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:06.922658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:06.924320 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
W20260928 02:42:06.924620 11714 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:06.919420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:39495 (local address 127.2.123.62:41405)
0928 02:42:06.919550 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:06.919554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:06.919588 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:42:06.919680 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:06.919682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:06.919721 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:06.919783 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:06.920269 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.920786 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.921583 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:06.921862 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:06.921986 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:06.924508 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:39495: BlockingRecv error: recv got EOF from 127.2.123.2:39495 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:42:07 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:07 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:07.110679 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:07.105117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:59379)
0928 02:42:07.105390 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:07.105403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:07.105545 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:07.105861 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:07.105864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:07.106067 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:07.106163 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:07.106169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.106984 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.106987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:07.107579 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:07.107587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.108088 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.108092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:07.108162 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:07.108635 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:07.108648 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:07.110519 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
W20260928 02:42:07.110754 11714 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:07.105229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:59379 (local address 127.2.123.62:41405)
0928 02:42:07.105383 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:07.105386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:07.105475 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:42:07.105672 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:07.105675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:07.105727 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:07.105809 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:07.106288 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.106841 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.107716 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.107953 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.108044 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:07.110653 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:59379: BlockingRecv error: recv got EOF from 127.2.123.3:59379 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 02:42:07 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:07 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:07.753485 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:07.748139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:38065)
0928 02:42:07.748409 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:07.748423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:07.748522 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:07.748754 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:07.748756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:07.748931 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:07.749024 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:07.749029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.749843 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.749846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:07.750566 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:07.750577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.751069 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.751072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:07.751164 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:07.751593 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:07.751603 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:07.753340 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
W20260928 02:42:07.753608 11718 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:07.748300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:38065 (local address 127.2.123.62:41405)
0928 02:42:07.748540 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:07.748544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:07.748559 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:42:07.748601 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:07.748603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:07.748643 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:07.748718 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:07.749161 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.749705 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.750682 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.750961 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.751049 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:07.753483 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:38065: BlockingRecv error: recv got EOF from 127.2.123.1:38065 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":64,"threads_started":1}
Sep 28 02:42:07 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:07 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:07.930920 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:07.925269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:44627)
0928 02:42:07.925528 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:07.925544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:07.925641 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:07.925929 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:07.925932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:07.926201 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:07.926315 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:07.926322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.927168 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.927172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:07.927899 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:07.927911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.928459 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.928463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:07.928545 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:07.929042 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:07.929056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:07.930805 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
W20260928 02:42:07.931108 11718 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:07.925355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:44627 (local address 127.2.123.62:41405)
0928 02:42:07.925486 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:07.925490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:07.925650 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:42:07.925701 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:07.925703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:07.925754 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:07.925828 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:07.926463 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.927025 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.928095 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:07.928354 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:07.928439 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:07.930975 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:44627: BlockingRecv error: recv got EOF from 127.2.123.2:44627 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:42:08 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:08 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:08.117059 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:08.111576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:38255)
0928 02:42:08.111882 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:08.111899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:08.112017 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:08.112262 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:08.112265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:08.112480 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:08.112570 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:08.112576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.113326 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.113330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:08.113900 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:08.113908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.114387 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.114391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:08.114474 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:08.114984 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:08.114997 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:08.116936 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
W20260928 02:42:08.117303 11718 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:08.111712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:38255 (local address 127.2.123.62:41405)
0928 02:42:08.111950 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:08.111955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:08.111970 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:42:08.112107 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:08.112110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:08.112160 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:08.112225 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:08.112694 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.113219 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.114045 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.114276 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.114364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:08.117192 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:38255: BlockingRecv error: recv got EOF from 127.2.123.3:38255 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 02:42:08 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:08 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:08.760207 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:08.754374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:53991)
0928 02:42:08.754689 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:08.754705 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:08.754807 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:08.755116 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:08.755118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:08.755295 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:08.755390 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:08.755395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.756397 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.756400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:08.757129 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:08.757140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.757665 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.757669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:08.757758 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:08.758274 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:08.758286 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:08.760091 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":73,"threads_started":1}
W20260928 02:42:08.760339 11722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:08.754471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:53991 (local address 127.2.123.62:41405)
0928 02:42:08.754827 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:08.754831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:08.754876 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:42:08.754924 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:08.754930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:08.754977 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:08.755058 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:08.755567 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.756253 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.757242 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.757516 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.757611 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:08.760220 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:53991: BlockingRecv error: recv got EOF from 127.2.123.1:53991 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":163,"threads_started":1}
Sep 28 02:42:08 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:08 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:08.937572 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:08.931697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:43799)
0928 02:42:08.931987 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:08.932001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:08.932112 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:08.932323 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:08.932325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:08.932562 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:08.932657 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:08.932663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.933423 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.933427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:08.934110 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:08.934119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.934649 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.934652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:08.934729 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:08.935281 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:08.935299 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:08.937443 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":68,"threads_started":1}
W20260928 02:42:08.937676 11722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:08.931796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:43799 (local address 127.2.123.62:41405)
0928 02:42:08.931973 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:08.931977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:08.932072 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:42:08.932186 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:08.932188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:08.932230 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:08.932301 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:08.932770 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.933306 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.934258 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:08.934537 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:08.934628 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:08.937578 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:43799: BlockingRecv error: recv got EOF from 127.2.123.2:43799 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 02:42:09 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:09 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:09.123664 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:09.117907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:46143)
0928 02:42:09.118248 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:09.118262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:09.118370 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:09.118639 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:09.118642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:09.118863 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:09.118965 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:09.118971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.119743 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.119746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:09.120316 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:09.120325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.120846 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.120849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:09.120920 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:09.121419 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:09.121431 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:09.123534 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":66,"threads_started":1}
W20260928 02:42:09.123817 11722 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:09.118051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:46143 (local address 127.2.123.62:41405)
0928 02:42:09.118202 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:09.118206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:09.118322 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:42:09.118443 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:09.118445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:09.118492 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:09.118561 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:09.119075 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.119626 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.120462 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.120695 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.120807 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:09.123697 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:46143: BlockingRecv error: recv got EOF from 127.2.123.3:46143 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:42:09 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:09 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:09.766716 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:09.760982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:44167)
0928 02:42:09.761277 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:09.761294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:09.761406 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:09.761689 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:09.761705 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:09.761877 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:09.761996 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:09.762003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.762812 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.762816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:09.763400 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:09.763410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.763889 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.763892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:09.763958 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:09.764545 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:09.764557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:09.766555 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
W20260928 02:42:09.766832 11726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:09.761159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:44167 (local address 127.2.123.62:41405)
0928 02:42:09.761483 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:09.761487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:09.761500 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:42:09.761544 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:09.761546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:09.761586 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:09.761665 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:09.762109 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.762689 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.763529 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.763781 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.763869 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:09.766715 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:44167: BlockingRecv error: recv got EOF from 127.2.123.1:44167 (error 108)
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
Sep 28 02:42:09 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:09 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:09.944121 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:09.938350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:39499)
0928 02:42:09.938613 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:09.938628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:09.938743 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:09.939055 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:09.939059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:09.939251 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:09.939344 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:09.939350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.940245 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.940248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:09.940804 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:09.940812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.941307 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.941311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:09.941395 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:09.941878 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:09.941890 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:09.943906 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
W20260928 02:42:09.944188 11726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:09.938423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:39499 (local address 127.2.123.62:41405)
0928 02:42:09.938595 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:09.938599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:09.938696 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:42:09.938850 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:09.938852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:09.938896 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:09.938957 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:09.939478 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.940142 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.940927 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:09.941177 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:09.941265 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:09.944084 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:39499: BlockingRecv error: recv got EOF from 127.2.123.2:39499 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 02:42:10 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:10 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:10.130617 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:10.124666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:45757)
0928 02:42:10.124938 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:10.124955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:10.125076 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:10.125316 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:10.125319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:10.125524 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:10.125618 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:10.125624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.126579 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.126582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:10.127141 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:10.127149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.127579 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.127582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:10.127675 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:10.128232 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:10.128249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:10.130462 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
W20260928 02:42:10.130695 11726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:10.124774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:45757 (local address 127.2.123.62:41405)
0928 02:42:10.124910 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:10.124915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:10.125032 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:42:10.125162 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:10.125165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:10.125214 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:10.125279 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:10.125733 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.126471 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.127262 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.127483 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.127566 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:10.130595 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:45757: BlockingRecv error: recv got EOF from 127.2.123.3:45757 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:42:10 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:10 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:10.778168 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:10.767533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:33649)
0928 02:42:10.767949 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:10.767975 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:10.768075 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:10.768358 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:10.768360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:10.768531 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:10.768686 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:10.768693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.769450 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.769453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:10.770148 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:10.770158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.770727 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.770730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:10.770797 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:10.776025 (+ 5228us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:10.776038 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:10.778037 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":163,"threads_started":1}
W20260928 02:42:10.778291 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:10.767627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:33649 (local address 127.2.123.62:41405)
0928 02:42:10.767875 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:10.767879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:10.768073 (+ 194us) server_negotiation.cc:407] Connection header received
0928 02:42:10.768146 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:10.768148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:10.768201 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:10.768281 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:10.768803 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.769337 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.770317 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.770545 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.770695 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:10.778179 (+ 7484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:33649: BlockingRecv error: recv got EOF from 127.2.123.1:33649 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
Sep 28 02:42:10 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:10 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:10.950568 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:10.945024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:48619)
0928 02:42:10.945281 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:10.945296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:10.945396 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:10.945652 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:10.945655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:10.945838 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:10.945934 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:10.945939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.946779 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.946783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:10.947360 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:10.947367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.947795 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.947799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:10.947879 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:10.948371 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:10.948383 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:10.950430 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
W20260928 02:42:10.950718 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:10.945158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:48619 (local address 127.2.123.62:41405)
0928 02:42:10.945350 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:10.945353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:10.945367 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:42:10.945493 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:10.945496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:10.945541 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:10.945608 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:10.946110 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.946667 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.947471 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:10.947693 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:10.947778 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:10.950592 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:48619: BlockingRecv error: recv got EOF from 127.2.123.2:48619 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 02:42:11 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:11 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:11.138360 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:11.131670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:52221)
0928 02:42:11.132047 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:11.132063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:11.132192 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:11.132644 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:11.132647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:11.132929 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:11.133050 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:11.133057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.134671 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.134675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:11.135316 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:11.135325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.135828 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.135832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:11.135904 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:11.136395 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:11.136406 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:11.138232 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
W20260928 02:42:11.138582 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:11.131824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:52221 (local address 127.2.123.62:41405)
0928 02:42:11.131979 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:11.131982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:11.132138 (+ 156us) server_negotiation.cc:407] Connection header received
0928 02:42:11.132325 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:11.132358 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:11.132506 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:11.132607 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:11.133208 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.134229 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.135451 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.135694 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.135782 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:11.138457 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:52221: BlockingRecv error: recv got EOF from 127.2.123.3:52221 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:42:11 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:11 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:11.784696 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:11.779014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:49783)
0928 02:42:11.779262 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:11.779276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:11.779357 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:11.779575 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:11.779577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:11.779772 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:11.779867 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:11.779873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.780650 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.780653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:11.781239 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:11.781249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.781667 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.781670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:11.781754 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:11.782261 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:11.782276 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:11.784493 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
W20260928 02:42:11.784757 11734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:11.779094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:49783 (local address 127.2.123.62:41405)
0928 02:42:11.779337 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:11.779341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:11.779354 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:42:11.779431 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:11.779434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:11.779473 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:11.779559 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:11.779970 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.780554 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.781356 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.781573 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.781652 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:11.784645 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:49783: BlockingRecv error: recv got EOF from 127.2.123.1:49783 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
Sep 28 02:42:11 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:11 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:11.956476 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:11.951297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:38951)
0928 02:42:11.951544 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:11.951557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:11.951651 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:11.951861 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:11.951863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:11.952032 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:11.952128 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:11.952176 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.952876 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.952879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:11.953478 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:11.953492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.954015 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.954018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:11.954082 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:11.954574 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:11.954586 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:11.956353 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
W20260928 02:42:11.956625 11734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:11.951388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:38951 (local address 127.2.123.62:41405)
0928 02:42:11.951519 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:11.951522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:11.951624 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:42:11.951721 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:11.951723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:11.951767 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:11.951846 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:11.952259 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.952780 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.953620 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:11.953864 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:11.953994 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:11.956501 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:38951: BlockingRecv error: recv got EOF from 127.2.123.2:38951 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:42:12 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:12 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:12.144537 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:12.139167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:38765)
0928 02:42:12.139416 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:12.139434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:12.139529 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:12.139743 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:12.139745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:12.139983 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:12.140085 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:12.140091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.140885 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.140889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:12.141510 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:12.141520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.142051 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.142054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:12.142121 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:12.142616 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:12.142627 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:12.144417 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
W20260928 02:42:12.144721 11734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:12.139323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:38765 (local address 127.2.123.62:41405)
0928 02:42:12.139474 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:12.139477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:12.139491 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:42:12.139609 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:12.139611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:12.139660 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:12.139727 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:12.140194 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.140725 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.141641 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.141869 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.142027 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:12.144566 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:38765: BlockingRecv error: recv got EOF from 127.2.123.3:38765 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 02:42:12 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:12 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:12.791129 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:12.785568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:45689)
0928 02:42:12.785899 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:12.785912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:12.786046 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:12.786350 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:12.786353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:12.786544 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:12.786656 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:12.786678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.787468 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.787471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:12.788028 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:12.788036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.788476 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.788479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:12.788557 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:12.789076 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:12.789088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:12.791005 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":74,"threads_started":1}
W20260928 02:42:12.791368 11737 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:12.785617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:45689 (local address 127.2.123.62:41405)
0928 02:42:12.785879 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:12.785883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:12.785979 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:42:12.786138 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:12.786140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:12.786186 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:12.786263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:12.786810 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.787356 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.788144 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.788376 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.788456 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:12.791214 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:45689: BlockingRecv error: recv got EOF from 127.2.123.1:45689 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
Sep 28 02:42:12 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:12 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:12.962646 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:12.957260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:53067)
0928 02:42:12.957509 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:12.957523 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:12.957655 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:12.957898 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:12.957901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:12.958137 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:12.958232 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:12.958238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.958974 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.958977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:12.959531 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:12.959538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.959952 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.959956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:12.960038 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:12.960513 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:12.960525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:12.962497 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
W20260928 02:42:12.962740 11737 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:12.957306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:53067 (local address 127.2.123.62:41405)
0928 02:42:12.957478 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:12.957482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:12.957655 (+ 173us) server_negotiation.cc:407] Connection header received
0928 02:42:12.957752 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:12.957755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:12.957805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:12.957870 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:12.958341 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.958877 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.959635 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:12.959855 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:12.959938 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:12.962645 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:53067: BlockingRecv error: recv got EOF from 127.2.123.2:53067 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 02:42:13 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:13 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:13.150712 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:13.145359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:59485)
0928 02:42:13.145673 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:13.145689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:13.145800 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:13.146074 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:13.146077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:13.146287 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:13.146404 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:13.146413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.147218 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.147221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:13.147822 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:13.147831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.148265 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.148268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:13.148327 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:13.148834 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:13.148846 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:13.150573 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":90,"threads_started":1}
W20260928 02:42:13.150820 11737 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:13.145515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:59485 (local address 127.2.123.62:41405)
0928 02:42:13.145652 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:13.145656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:13.145762 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:42:13.145874 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:13.145877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:13.145939 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:13.146057 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:13.146585 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.147129 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.147939 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.148166 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.148250 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:13.150721 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:59485: BlockingRecv error: recv got EOF from 127.2.123.3:59485 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:42:13 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:13 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:13.797183 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:13.791957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:49741)
0928 02:42:13.792339 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:13.792353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:13.792450 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:13.792677 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:13.792679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:13.792862 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:13.792955 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:13.792960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.793821 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.793825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:13.794483 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:13.794494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.794939 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.794942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:13.795007 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:13.795462 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:13.795474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:13.797063 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":89,"threads_started":1}
W20260928 02:42:13.797338 11742 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:13.792058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:49741 (local address 127.2.123.62:41405)
0928 02:42:13.792364 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:13.792367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:13.792413 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:42:13.792538 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:13.792540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:13.792582 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:13.792650 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:13.793042 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.793688 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.794599 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.794833 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.794907 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:13.797226 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:49741: BlockingRecv error: recv got EOF from 127.2.123.1:49741 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":129,"threads_started":1}
Sep 28 02:42:13 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:13 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:13.968782 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:13.963402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:39813)
0928 02:42:13.963737 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:13.963752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:13.963859 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:13.964150 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:13.964153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:13.964337 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:13.964427 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:13.964433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.965175 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.965179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:13.965731 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:13.965739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.966219 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.966222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:13.966304 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:13.966738 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:13.966748 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:13.968635 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
W20260928 02:42:13.968835 11742 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:13.963506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:39813 (local address 127.2.123.62:41405)
0928 02:42:13.963682 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:13.963687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:13.963822 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:42:13.963950 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:13.963953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:13.963999 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:13.964064 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:13.964544 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.965076 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.965854 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:13.966108 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:13.966189 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:13.968741 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:39813: BlockingRecv error: recv got EOF from 127.2.123.2:39813 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:42:14 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:14 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:14.157670 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:14.151723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:40581)
0928 02:42:14.152050 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:14.152065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:14.152186 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:14.152468 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:14.152470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:14.152645 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:14.152742 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:14.152748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.153577 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.153580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:14.154170 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:14.154184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.154677 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.154680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:14.154749 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:14.155283 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:14.155296 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:14.157522 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":78,"threads_started":1}
W20260928 02:42:14.157787 11742 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:14.151836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:40581 (local address 127.2.123.62:41405)
0928 02:42:14.151989 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:14.151993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:14.152183 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:42:14.152274 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:14.152277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:14.152328 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:14.152403 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:14.152877 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.153452 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.154284 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.154538 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.154625 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:14.157673 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:40581: BlockingRecv error: recv got EOF from 127.2.123.3:40581 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:42:14 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:14 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:14.803184 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:14.798033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:38397)
0928 02:42:14.798301 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:14.798316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:14.798421 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:14.798683 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:14.798686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:14.798877 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:14.798966 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:14.798972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.799708 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.799712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:14.800293 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:14.800302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.800728 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.800732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:14.800810 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:14.801294 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:14.801307 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:14.803014 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
W20260928 02:42:14.803280 11746 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:14.798119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:38397 (local address 127.2.123.62:41405)
0928 02:42:14.798392 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:14.798396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:14.798409 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:42:14.798526 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:14.798529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:14.798575 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:14.798644 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:14.799075 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.799607 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.800405 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.800633 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.800721 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:14.803166 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:38397: BlockingRecv error: recv got EOF from 127.2.123.1:38397 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
Sep 28 02:42:14 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:14 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:14.975178 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:14.969644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:48159)
0928 02:42:14.969906 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:14.969918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:14.970057 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:14.970345 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:14.970349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:14.970552 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:14.970669 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:14.970677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.971374 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.971376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:14.971928 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:14.971935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.972410 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.972414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:14.972485 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:14.972966 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:14.972979 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:14.975041 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
W20260928 02:42:14.975328 11746 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:14.969744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:48159 (local address 127.2.123.62:41405)
0928 02:42:14.969876 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:14.969880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:14.970005 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:42:14.970159 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:14.970162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:14.970217 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:14.970306 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:14.970767 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.971291 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.972032 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:14.972258 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:14.972338 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:14.975201 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:48159: BlockingRecv error: recv got EOF from 127.2.123.2:48159 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:42:15 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:15 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:15.163794 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:15.158460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:36523)
0928 02:42:15.158725 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:15.158743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:15.158847 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:15.159055 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:15.159057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:15.159251 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:15.159347 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:15.159354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.160145 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.160148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:15.160732 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:15.160741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.161219 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.161222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:15.161292 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:15.161793 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:15.161805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:15.163671 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
W20260928 02:42:15.163903 11746 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:15.158532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:36523 (local address 127.2.123.62:41405)
0928 02:42:15.158657 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:15.158660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:15.158801 (+ 141us) server_negotiation.cc:407] Connection header received
0928 02:42:15.158920 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:15.158922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:15.158974 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:15.159040 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:15.159481 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.160024 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.160848 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.161098 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.161180 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:15.163804 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:36523: BlockingRecv error: recv got EOF from 127.2.123.3:36523 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:42:15 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:15 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:15.810568 11749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:15.803960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:41667)
0928 02:42:15.804326 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:15.804380 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:15.804493 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:15.804795 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:15.804798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:15.805030 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:15.805147 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:15.805154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.806081 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.806084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:15.806887 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:15.806900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.807556 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.807560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:15.807666 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:15.808186 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:15.808202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:15.810403 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":81,"threads_started":1}
W20260928 02:42:15.810807 11750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:15.804049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:41667 (local address 127.2.123.62:41405)
0928 02:42:15.804417 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:15.804421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:15.804438 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:42:15.804625 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:15.804629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:15.804677 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:15.804752 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:15.805255 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.805912 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.807006 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.807420 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.807528 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:15.810683 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:41667: BlockingRecv error: recv got EOF from 127.2.123.1:41667 (error 108)
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":73,"threads_started":1}
Sep 28 02:42:15 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:15 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:15.981065 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:15.975858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:36621)
0928 02:42:15.976109 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:15.976123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:15.976270 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:15.976487 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:15.976489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:15.976666 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:15.976759 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:15.976764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.977581 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.977584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:15.978197 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:15.978206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.978625 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.978627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:15.978691 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:15.979225 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:15.979237 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:15.980941 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
W20260928 02:42:15.981189 11750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:15.975956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:36621 (local address 127.2.123.62:41405)
0928 02:42:15.976172 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:15.976181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:15.976219 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:42:15.976349 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:15.976351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:15.976394 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:15.976457 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:15.976885 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.977468 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.978307 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:15.978531 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:15.978612 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:15.981067 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:36621: BlockingRecv error: recv got EOF from 127.2.123.2:36621 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 02:42:16 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:16 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.3@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:16.169903 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:16.164491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.3:40705)
0928 02:42:16.164762 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:16.164776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:16.164872 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:16.165105 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:16.165107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:16.165286 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:16.165382 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:16.165388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.166271 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.166275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:16.166886 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:16.166897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.167427 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.167430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:16.167502 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:16.167965 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:16.167977 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:16.169787 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
W20260928 02:42:16.170133 11750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:16.164553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:40705 (local address 127.2.123.62:41405)
0928 02:42:16.164664 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:16.164667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:16.164873 (+ 206us) server_negotiation.cc:407] Connection header received
0928 02:42:16.164939 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:16.164941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:16.164990 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:16.165074 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:16.165502 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.166160 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.167045 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.167315 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.167400 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:16.169929 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.3:40705: BlockingRecv error: recv got EOF from 127.2.123.3:40705 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 02:42:16 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:16 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.1@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:16.816629 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:16.811496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.1:38693)
0928 02:42:16.811764 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:16.811778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:16.811858 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:16.812067 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:16.812069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:16.812253 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:16.812345 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:16.812350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.813097 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.813100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:16.813659 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:16.813669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.814183 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.814187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:16.814266 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:16.814711 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:16.814726 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:16.816518 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
W20260928 02:42:16.816764 11754 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:16.811545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:38693 (local address 127.2.123.62:41405)
0928 02:42:16.811805 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:16.811809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:16.811822 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:42:16.811931 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:16.811933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:16.811973 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:16.812045 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:16.812436 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.812981 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.813780 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.814060 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.814161 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:16.816649 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.1:38693: BlockingRecv error: recv got EOF from 127.2.123.1:38693 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
Sep 28 02:42:16 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:42:16 dist-test-slave-rqvb krb5kdc[10696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.123.2@KRBTEST.COM for oryx/127.2.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:42:16.987572 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:16.982056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41405 (local address 127.2.123.2:38515)
0928 02:42:16.982325 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:16.982338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:16.982430 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:16.982746 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:16.982748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:16.982916 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:16.983006 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:16.983011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.983804 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.983807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:16.984374 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:16.984382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.984878 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.984881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:16.984951 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:16.985411 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:16.985422 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:16.987445 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.123.62:41405: Server oryx/127.2.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
W20260928 02:42:16.987783 11754 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:16.982173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:38515 (local address 127.2.123.62:41405)
0928 02:42:16.982307 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:16.982311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:16.982402 (+ 91us) server_negotiation.cc:407] Connection header received
0928 02:42:16.982503 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:16.982506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:16.982627 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:16.982716 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:16.983133 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.983663 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.984494 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:16.984734 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:16.984823 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:16.987609 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.123.2:38515: BlockingRecv error: recv got EOF from 127.2.123.2:38515 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:42:17.055666 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 11279
I20260928 02:42:17.060972 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 11431
I20260928 02:42:17.065927 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 11562
I20260928 02:42:17.070375 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 11209
2026-09-28T02:42:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17839 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:17Z Disabled control of system clock
I20260928 02:42:17.098405 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:37805
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:42:17.206965 11759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:17.207255 11759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:17.207288 11759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:17.211236 11759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:17.211292 11759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:17.211305 11759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:17.211320 11759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:17.211334 11759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:17.216207 11759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:37805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:17.217239 11759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:17.218600 11759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:17.227130 11765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:17.227154 11764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:17.227228 11759 server_base.cc:1034] running on GCE node
W20260928 02:42:17.227144 11767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:17.227957 11759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:17.229017 11759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:17.230158 11759 hybrid_clock.cc:648] HybridClock initialized: now 1790563337230132 us; error 38 us; skew 500 ppm
I20260928 02:42:17.232005 11759 webserver.cc:460] Webserver started at http://127.2.123.62:32771/ using document root <none> and password file <none>
I20260928 02:42:17.232548 11759 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:17.232609 11759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:17.232800 11759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:17.234848 11759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "24c490beb86f4560a9d4f30c2f7f517c"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "e3b8140dce55f5cb3a09b239ed694759"
server_key_iv: "526d1c9077f78159958c22f27b0723a7"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.235370 11759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "24c490beb86f4560a9d4f30c2f7f517c"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "e3b8140dce55f5cb3a09b239ed694759"
server_key_iv: "526d1c9077f78159958c22f27b0723a7"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.239013 11759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:17.241460 11773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.242790 11759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:42:17.242900 11759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "24c490beb86f4560a9d4f30c2f7f517c"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "e3b8140dce55f5cb3a09b239ed694759"
server_key_iv: "526d1c9077f78159958c22f27b0723a7"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.243029 11759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:17.267434 11759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:17.268183 11759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:17.268359 11759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:17.274955 11759 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:37805
I20260928 02:42:17.274977 11825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:37805 every 8 connection(s)
I20260928 02:42:17.275872 11759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:17.279126 11826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:17.284075 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 11759
I20260928 02:42:17.284188 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:42:17.284440 2540 external_mini_cluster.cc:1428] Setting key c9923e27e47fdfe110239813c7436d73
I20260928 02:42:17.284852 11826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c: Bootstrap starting.
I20260928 02:42:17.287570 11826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:17.288460 11826 log.cc:824] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:17.290638 11826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c: No bootstrap required, opened a new log
I20260928 02:42:17.295008 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:17.285925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.2.123.62:37805)
0928 02:42:17.286664 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:17.286682 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:17.286725 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:42:17.287544 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:17.287568 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:17.288098 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:17.288464 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:17.288920 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.290271 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.291252 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.291931 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.292090 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:17.292812 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:17.292819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:17.292835 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:17.293347 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:17.293415 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:17.293440 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:17.293541 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:17.293704 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:42:17.294042 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":116,"threads_started":1}
I20260928 02:42:17.294916 11826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c490beb86f4560a9d4f30c2f7f517c" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37805 } }
I20260928 02:42:17.295185 11826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:17.295266 11826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c490beb86f4560a9d4f30c2f7f517c, State: Initialized, Role: FOLLOWER
I20260928 02:42:17.295655 11826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c490beb86f4560a9d4f30c2f7f517c" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37805 } }
I20260928 02:42:17.295791 11826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:17.295851 11826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:17.295945 11826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:17.297037 11826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c490beb86f4560a9d4f30c2f7f517c" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37805 } }
I20260928 02:42:17.297405 11826 leader_election.cc:302] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24c490beb86f4560a9d4f30c2f7f517c; no voters:
I20260928 02:42:17.297683 11826 leader_election.cc:288] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:17.297780 11831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:17.298030 11831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [term 1 LEADER]: Becoming Leader. State: Replica: 24c490beb86f4560a9d4f30c2f7f517c, State: Running, Role: LEADER
I20260928 02:42:17.298347 11831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c490beb86f4560a9d4f30c2f7f517c" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37805 } }
I20260928 02:42:17.298839 11826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:17.299841 11832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24c490beb86f4560a9d4f30c2f7f517c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c490beb86f4560a9d4f30c2f7f517c" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37805 } } }
I20260928 02:42:17.299958 11832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:17.300376 11833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c490beb86f4560a9d4f30c2f7f517c. Latest consensus state: current_term: 1 leader_uuid: "24c490beb86f4560a9d4f30c2f7f517c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c490beb86f4560a9d4f30c2f7f517c" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37805 } } }
I20260928 02:42:17.300459 11840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:17.300572 11833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:17.303335 11840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:17.308874 11840 catalog_manager.cc:1347] Generated new cluster ID: 803ac505515a436c8f0261f1134ebba8
I20260928 02:42:17.308931 11840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:17.320392 11840 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:17.321370 11840 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:17.332720 11840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24c490beb86f4560a9d4f30c2f7f517c: Generated new TSK 0
I20260928 02:42:17.333462 11840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:17.337481 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:37805
--builtin_ntp_servers=127.2.123.20:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:42:17.487169 11850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:17.487486 11850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:17.487527 11850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:17.491539 11850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:17.491595 11850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:17.491647 11850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:42:17.496341 11850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:17.497296 11850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:17.498622 11850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:17.508060 11855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:17.508065 11856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:17.508211 11850 server_base.cc:1034] running on GCE node
W20260928 02:42:17.508080 11858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:17.508749 11850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:17.509357 11850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:17.510530 11850 hybrid_clock.cc:648] HybridClock initialized: now 1790563337510509 us; error 32 us; skew 500 ppm
I20260928 02:42:17.512245 11850 webserver.cc:460] Webserver started at http://127.2.123.1:40639/ using document root <none> and password file <none>
I20260928 02:42:17.512830 11850 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:17.512890 11850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:17.513072 11850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:17.515125 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "f5ffa37dff09459988f459105ffd68bd"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "1c942dc0a401f2afaaac53c97e65c13b"
server_key_iv: "1addd4bb72a02b76e0cdd0b3a4706b03"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.515645 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "f5ffa37dff09459988f459105ffd68bd"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "1c942dc0a401f2afaaac53c97e65c13b"
server_key_iv: "1addd4bb72a02b76e0cdd0b3a4706b03"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.519189 11850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:42:17.521786 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.523082 11850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:17.523186 11850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "f5ffa37dff09459988f459105ffd68bd"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "1c942dc0a401f2afaaac53c97e65c13b"
server_key_iv: "1addd4bb72a02b76e0cdd0b3a4706b03"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.523296 11850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:17.544988 11850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:17.545668 11850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:17.545835 11850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:17.546474 11850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:17.547452 11850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:17.547498 11850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.547545 11850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:17.547574 11850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.555966 11850 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:42733
I20260928 02:42:17.555991 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:42733 every 8 connection(s)
I20260928 02:42:17.556842 11850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:17.564944 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 11850
I20260928 02:42:17.565059 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:17.565367 2540 external_mini_cluster.cc:1428] Setting key 36be07ea8e2bd885808679e3544feb11
I20260928 02:42:17.565837 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:17.558519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:47069 (local address 127.2.123.62:37805)
0928 02:42:17.558636 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:17.558640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:17.559688 (+ 1048us) server_negotiation.cc:407] Connection header received
0928 02:42:17.560985 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:17.560988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:17.561036 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:17.561108 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:17.561980 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.562544 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.563685 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.563927 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.564039 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:17.565252 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:17.565256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:17.565259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:17.565274 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:17.565298 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:17.565308 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:17.565427 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:17.565630 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:42:17.565692 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:42:17.566792 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:17.558932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37805 (local address 127.2.123.1:47069)
0928 02:42:17.559540 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:17.559579 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:17.560764 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:17.561252 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:17.561259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:17.561344 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:17.561812 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:17.561823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.562652 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.562655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:17.563557 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:17.563566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.564179 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.564184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:17.564329 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:17.564963 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:17.564964 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:17.565005 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:17.565084 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:17.565470 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:17.565481 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:17.565488 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:42:17.565604 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:42:17.565869 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":109,"threads_started":1}
I20260928 02:42:17.567255 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:37805
--builtin_ntp_servers=127.2.123.20:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:42:17.567998 11978 heartbeater.cc:347] Connected to a master server at 127.2.123.62:37805
I20260928 02:42:17.568228 11978 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:17.568769 11978 heartbeater.cc:511] Master 127.2.123.62:37805 requested a full tablet report, sending...
I20260928 02:42:17.570242 11788 ts_manager.cc:194] Registered new tserver with Master: f5ffa37dff09459988f459105ffd68bd (127.2.123.1:42733)
I20260928 02:42:17.571385 11788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.1:47069
W20260928 02:42:17.674111 11981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:17.674360 11981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:17.674402 11981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:17.678359 11981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:17.678418 11981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:17.678470 11981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:42:17.683135 11981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:17.684159 11981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:17.685490 11981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:17.695096 11986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:17.695185 11989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:17.695204 11987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:17.695413 11981 server_base.cc:1034] running on GCE node
I20260928 02:42:17.695807 11981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:17.696481 11981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:17.697647 11981 hybrid_clock.cc:648] HybridClock initialized: now 1790563337697632 us; error 29 us; skew 500 ppm
I20260928 02:42:17.699630 11981 webserver.cc:460] Webserver started at http://127.2.123.2:37689/ using document root <none> and password file <none>
I20260928 02:42:17.700168 11981 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:17.700253 11981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:17.700455 11981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:17.702487 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "65412e4a53f04b1db1a6a534498dc8f6"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "7e2bbed02686545f985b20f0e6a6f185"
server_key_iv: "e5604d2eb6da09ee176f5a85807eea61"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.703018 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "65412e4a53f04b1db1a6a534498dc8f6"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "7e2bbed02686545f985b20f0e6a6f185"
server_key_iv: "e5604d2eb6da09ee176f5a85807eea61"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.706808 11981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:42:17.709206 11995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.710445 11981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:17.710547 11981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "65412e4a53f04b1db1a6a534498dc8f6"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "7e2bbed02686545f985b20f0e6a6f185"
server_key_iv: "e5604d2eb6da09ee176f5a85807eea61"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.710638 11981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:17.733582 11981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:17.734396 11981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:17.734587 11981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:17.735208 11981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:17.736222 11981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:17.736275 11981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.736332 11981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:17.736361 11981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.745772 11981 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:41343
I20260928 02:42:17.745860 12108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:41343 every 8 connection(s)
I20260928 02:42:17.746837 11981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:17.753683 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 11981
I20260928 02:42:17.753791 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:17.754086 2540 external_mini_cluster.cc:1428] Setting key 540194fa0cac7e75b2710adacc8cdbaf
I20260928 02:42:17.756037 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:37805
--builtin_ntp_servers=127.2.123.20:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:42:17.756142 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:17.748791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:39525 (local address 127.2.123.62:37805)
0928 02:42:17.748909 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:17.748913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:17.750024 (+ 1111us) server_negotiation.cc:407] Connection header received
0928 02:42:17.751422 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:17.751425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:17.751479 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:17.751553 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:17.752432 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.752955 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.754132 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.754374 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.754464 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:17.755708 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:17.755712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:17.755714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:17.755724 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:17.755743 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:17.755750 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:17.755805 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:17.755944 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:42:17.756002 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:42:17.756979 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:17.749157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37805 (local address 127.2.123.2:39525)
0928 02:42:17.749855 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:17.749897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:17.751205 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:17.751725 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:17.751732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:17.751815 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:17.752287 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:17.752298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.753068 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.753071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:17.754020 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:17.754029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.754511 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.754515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:17.754709 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:17.755475 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:17.755477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:17.755517 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:17.755577 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:17.755815 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:17.755824 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:17.755829 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:42:17.755937 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:42:17.756251 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260928 02:42:17.758066 12109 heartbeater.cc:347] Connected to a master server at 127.2.123.62:37805
I20260928 02:42:17.758347 12109 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:17.758939 12109 heartbeater.cc:511] Master 127.2.123.62:37805 requested a full tablet report, sending...
I20260928 02:42:17.760032 11788 ts_manager.cc:194] Registered new tserver with Master: 65412e4a53f04b1db1a6a534498dc8f6 (127.2.123.2:41343)
I20260928 02:42:17.760617 11788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.2:39525
W20260928 02:42:17.864614 12112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:17.864897 12112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:17.864950 12112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:17.869012 12112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:17.869079 12112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:17.869154 12112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:42:17.874112 12112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:17.875190 12112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:17.876976 12112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:17.887655 12120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:17.887661 12117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:17.887735 12112 server_base.cc:1034] running on GCE node
W20260928 02:42:17.887713 12118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:17.888325 12112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:17.888948 12112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:17.890095 12112 hybrid_clock.cc:648] HybridClock initialized: now 1790563337890071 us; error 36 us; skew 500 ppm
I20260928 02:42:17.892063 12112 webserver.cc:460] Webserver started at http://127.2.123.3:45543/ using document root <none> and password file <none>
I20260928 02:42:17.892606 12112 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:17.892675 12112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:17.892866 12112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:17.894944 12112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "f5ae5ba1c8304ec48ab4168f85c57133"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "080e9046ed4a4ebcc56e2c3c2c059d49"
server_key_iv: "dd116d3efdb1be38a3b49eff8d354cb6"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.895468 12112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "f5ae5ba1c8304ec48ab4168f85c57133"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "080e9046ed4a4ebcc56e2c3c2c059d49"
server_key_iv: "dd116d3efdb1be38a3b49eff8d354cb6"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.899266 12112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:42:17.902191 12126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.903469 12112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:17.903578 12112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "f5ae5ba1c8304ec48ab4168f85c57133"
format_stamp: "Formatted at 2026-09-28 02:42:17 on dist-test-slave-rqvb"
server_key: "080e9046ed4a4ebcc56e2c3c2c059d49"
server_key_iv: "dd116d3efdb1be38a3b49eff8d354cb6"
server_key_version: "encryptionkey@0"
I20260928 02:42:17.903677 12112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:17.921556 12112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:17.922252 12112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:17.922420 12112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:17.922950 12112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:17.923832 12112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:17.923889 12112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.923939 12112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:17.923967 12112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:17.932230 12112 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:32881
I20260928 02:42:17.932391 12239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:32881 every 8 connection(s)
I20260928 02:42:17.933107 12112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:17.941870 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:17.934925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:58247 (local address 127.2.123.62:37805)
0928 02:42:17.935063 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:17.935067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:17.936178 (+ 1111us) server_negotiation.cc:407] Connection header received
0928 02:42:17.937555 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:17.937558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:17.937607 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:17.937703 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:17.938547 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.939087 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.940209 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.940430 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.940509 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:17.941466 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:17.941469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:17.941471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:17.941483 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:17.941501 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:17.941508 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:17.941563 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:17.941694 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:42:17.941747 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:42:17.942683 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:17.935415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37805 (local address 127.2.123.3:58247)
0928 02:42:17.936038 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:17.936077 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:17.937297 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:17.937812 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:17.937820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:17.937903 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:17.938414 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:17.938426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.939194 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.939197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:17.940087 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:17.940095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:17.940560 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:17.940563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:17.940705 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:17.941221 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:17.941222 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:17.941259 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:17.941327 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:17.941570 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:17.941574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:17.941578 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:42:17.941659 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:42:17.941867 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260928 02:42:17.942809 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 12112
I20260928 02:42:17.942904 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:17.943181 2540 external_mini_cluster.cc:1428] Setting key 2224ba6cc7606496ef440616062fb763
I20260928 02:42:17.943899 12240 heartbeater.cc:347] Connected to a master server at 127.2.123.62:37805
I20260928 02:42:17.944132 12240 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:17.944649 12240 heartbeater.cc:511] Master 127.2.123.62:37805 requested a full tablet report, sending...
I20260928 02:42:17.945582 11788 ts_manager.cc:194] Registered new tserver with Master: f5ae5ba1c8304ec48ab4168f85c57133 (127.2.123.3:32881)
I20260928 02:42:17.946192 11788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.3:58247
I20260928 02:42:17.956735 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:42:17.959388 11829 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:17.958551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.2.123.62:37805)
0928 02:42:17.958678 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:17.958683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:17.958850 (+ 167us) server_negotiation.cc:407] Connection header received
0928 02:42:17.958921 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:17.958923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:17.958965 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:17.959029 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:17.959255 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59782: BlockingRecv error: recv got EOF from 127.0.0.1:59782 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:42:17.959534 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 11850
I20260928 02:42:17.964243 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 11981
I20260928 02:42:17.968499 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 12112
I20260928 02:42:17.972311 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 11759
2026-09-28T02:42:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (900 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:17Z Disabled control of system clock
I20260928 02:42:17.999089 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:36457
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:36457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:42:18.104777 12251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:18.105043 12251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:18.105078 12251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:18.109061 12251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:18.109119 12251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:18.109133 12251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:18.109148 12251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:18.109162 12251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:18.113871 12251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:36457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:36457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:18.114924 12251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:18.116299 12251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:42:18.124991 12251 server_base.cc:1034] running on GCE node
W20260928 02:42:18.124917 12259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:18.124918 12257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:18.125209 12256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:18.125777 12251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:18.126889 12251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:18.128085 12251 hybrid_clock.cc:648] HybridClock initialized: now 1790563338128059 us; error 39 us; skew 500 ppm
I20260928 02:42:18.130162 12251 webserver.cc:460] Webserver started at http://127.2.123.62:39423/ using document root <none> and password file <none>
I20260928 02:42:18.130753 12251 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:18.130826 12251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:18.131040 12251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:18.133152 12251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "4f5579afc763474b9f10091258ef9254"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "dc9a13eb27f49869c7f8536457ed0290"
server_key_iv: "1ec2467a9bd0a72dfbb9412e1ecd8da4"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.133746 12251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "4f5579afc763474b9f10091258ef9254"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "dc9a13eb27f49869c7f8536457ed0290"
server_key_iv: "1ec2467a9bd0a72dfbb9412e1ecd8da4"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.137645 12251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:42:18.140288 12265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.141610 12251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:18.141737 12251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "4f5579afc763474b9f10091258ef9254"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "dc9a13eb27f49869c7f8536457ed0290"
server_key_iv: "1ec2467a9bd0a72dfbb9412e1ecd8da4"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.141849 12251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:18.146433 12251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:18.147018 12251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:18.147171 12251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:18.153719 12251 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:36457
I20260928 02:42:18.153759 12317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:36457 every 8 connection(s)
I20260928 02:42:18.154683 12251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:18.159031 12318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:18.164644 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 12251
I20260928 02:42:18.164759 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:42:18.165105 2540 external_mini_cluster.cc:1428] Setting key f6b039c10ddeb243edd2794e7dc728ba
I20260928 02:42:18.166391 12318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254: Bootstrap starting.
I20260928 02:42:18.168735 12318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:18.169590 12318 log.cc:824] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:18.170756 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:18.166671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.2.123.62:36457)
0928 02:42:18.167282 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:18.167298 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:18.167333 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:42:18.167959 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:18.167975 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:18.168334 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:18.168609 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:18.169374 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:18.169379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:18.169391 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:18.169401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:18.169463 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:18.169479 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:18.169569 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:18.169696 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:42:18.170081 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260928 02:42:18.171579 12318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254: No bootstrap required, opened a new log
I20260928 02:42:18.175482 12318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5579afc763474b9f10091258ef9254" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36457 } }
I20260928 02:42:18.175776 12318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:18.175851 12318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5579afc763474b9f10091258ef9254, State: Initialized, Role: FOLLOWER
I20260928 02:42:18.176327 12318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5579afc763474b9f10091258ef9254" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36457 } }
I20260928 02:42:18.176479 12318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:18.176559 12318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:18.176666 12318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:18.177630 12318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5579afc763474b9f10091258ef9254" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36457 } }
I20260928 02:42:18.178001 12318 leader_election.cc:302] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f5579afc763474b9f10091258ef9254; no voters:
I20260928 02:42:18.178315 12318 leader_election.cc:288] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:18.178447 12323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:18.178730 12323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [term 1 LEADER]: Becoming Leader. State: Replica: 4f5579afc763474b9f10091258ef9254, State: Running, Role: LEADER
I20260928 02:42:18.179019 12323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5579afc763474b9f10091258ef9254" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36457 } }
I20260928 02:42:18.179558 12318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:18.180563 12324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f5579afc763474b9f10091258ef9254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5579afc763474b9f10091258ef9254" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36457 } } }
I20260928 02:42:18.180783 12324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:18.181030 12325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f5579afc763474b9f10091258ef9254. Latest consensus state: current_term: 1 leader_uuid: "4f5579afc763474b9f10091258ef9254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f5579afc763474b9f10091258ef9254" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36457 } } }
I20260928 02:42:18.181138 12325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:18.181316 12332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:18.184018 12332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:18.189689 12332 catalog_manager.cc:1347] Generated new cluster ID: cf22e294ff1a466a988ecb5634d510ff
I20260928 02:42:18.189778 12332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:18.215656 12332 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:18.216007 12332 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:18.220887 12332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f5579afc763474b9f10091258ef9254: Generated new TSK 0
I20260928 02:42:18.221692 12332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:42:18.235594 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36457
--builtin_ntp_servers=127.2.123.20:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:42:18.344734 12342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:18.345002 12342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:18.345047 12342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:18.349090 12342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:18.349150 12342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:18.349207 12342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:42:18.353916 12342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:18.354978 12342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:18.356326 12342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:18.365744 12347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:18.365821 12342 server_base.cc:1034] running on GCE node
W20260928 02:42:18.365741 12350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:18.365741 12348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:18.366354 12342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:18.366954 12342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:18.368124 12342 hybrid_clock.cc:648] HybridClock initialized: now 1790563338368107 us; error 35 us; skew 500 ppm
I20260928 02:42:18.369874 12342 webserver.cc:460] Webserver started at http://127.2.123.1:46183/ using document root <none> and password file <none>
I20260928 02:42:18.370445 12342 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:18.370509 12342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:18.370695 12342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:18.372680 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "e7e9164145a241479d5fcc41177e4f01"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "5b8a7bcbc8814350a9c97c03e18e2455"
server_key_iv: "5e818babe9d55fc39cc71805909a2f98"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.373162 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "e7e9164145a241479d5fcc41177e4f01"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "5b8a7bcbc8814350a9c97c03e18e2455"
server_key_iv: "5e818babe9d55fc39cc71805909a2f98"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.376763 12342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 02:42:18.379339 12356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.380791 12342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:42:18.380914 12342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "e7e9164145a241479d5fcc41177e4f01"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "5b8a7bcbc8814350a9c97c03e18e2455"
server_key_iv: "5e818babe9d55fc39cc71805909a2f98"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.381083 12342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:18.385735 12342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:18.386358 12342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:18.386507 12342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:18.387006 12342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:18.387850 12342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:18.387893 12342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.387928 12342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:18.387966 12342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.396021 12342 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:43815
I20260928 02:42:18.396047 12469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:43815 every 8 connection(s)
I20260928 02:42:18.396970 12342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:18.401842 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 12342
I20260928 02:42:18.401937 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:18.402213 2540 external_mini_cluster.cc:1428] Setting key 71a051e1e2ab697a83e35629cba40e7f
I20260928 02:42:18.402861 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:18.398758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:43195 (local address 127.2.123.62:36457)
0928 02:42:18.398904 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:18.398908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:18.399946 (+ 1038us) server_negotiation.cc:407] Connection header received
0928 02:42:18.401216 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:18.401220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:18.401263 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:18.401337 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:18.402434 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:18.402437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:18.402440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:18.402452 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:18.402473 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:18.402481 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:18.402562 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:18.402691 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:42:18.402734 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:42:18.403690 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:18.399153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:36457 (local address 127.2.123.1:43195)
0928 02:42:18.399806 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:18.399845 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:18.400994 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:18.401504 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:18.401510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:18.401605 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:18.402170 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:18.402171 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:18.402213 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:18.402290 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:18.402560 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:18.402567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:18.402573 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:42:18.402729 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 02:42:18.402956 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260928 02:42:18.404125 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36457
--builtin_ntp_servers=127.2.123.20:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:42:18.404812 12470 heartbeater.cc:347] Connected to a master server at 127.2.123.62:36457
I20260928 02:42:18.405045 12470 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:18.405503 12470 heartbeater.cc:511] Master 127.2.123.62:36457 requested a full tablet report, sending...
I20260928 02:42:18.407132 12282 ts_manager.cc:194] Registered new tserver with Master: e7e9164145a241479d5fcc41177e4f01 (127.2.123.1:43815)
W20260928 02:42:18.513363 12473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:18.513620 12473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:18.513659 12473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:18.517511 12473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:18.517560 12473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:18.517612 12473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:42:18.522207 12473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:18.523168 12473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:18.524447 12473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:18.533865 12479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:18.533952 12473 server_base.cc:1034] running on GCE node
W20260928 02:42:18.533859 12478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:18.533859 12481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:18.534454 12473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:18.535135 12473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:18.536320 12473 hybrid_clock.cc:648] HybridClock initialized: now 1790563338536299 us; error 30 us; skew 500 ppm
I20260928 02:42:18.538200 12473 webserver.cc:460] Webserver started at http://127.2.123.2:36185/ using document root <none> and password file <none>
I20260928 02:42:18.538759 12473 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:18.538825 12473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:18.539024 12473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:18.541075 12473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "d4d4e06888c64b32a7e68b67f7b250b0"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "2c9129e0e4b6620888edc60880bfbe69"
server_key_iv: "656dcca0adc3f0d71d3dbb181b5b4c3b"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.541590 12473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "d4d4e06888c64b32a7e68b67f7b250b0"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "2c9129e0e4b6620888edc60880bfbe69"
server_key_iv: "656dcca0adc3f0d71d3dbb181b5b4c3b"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.545269 12473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:42:18.547909 12487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.549260 12473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:42:18.549373 12473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "d4d4e06888c64b32a7e68b67f7b250b0"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "2c9129e0e4b6620888edc60880bfbe69"
server_key_iv: "656dcca0adc3f0d71d3dbb181b5b4c3b"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.549481 12473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:18.554064 12473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:18.554663 12473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:18.554821 12473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:18.555348 12473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:18.556226 12473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:18.556270 12473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.556319 12473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:18.556348 12473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.564543 12473 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:33147
I20260928 02:42:18.564584 12600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:33147 every 8 connection(s)
I20260928 02:42:18.565392 12473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:18.570654 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 12473
I20260928 02:42:18.570767 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:18.571069 2540 external_mini_cluster.cc:1428] Setting key 06bb03cace9c4822a2c7ec22aa959443
I20260928 02:42:18.571462 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:18.567248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:47779 (local address 127.2.123.62:36457)
0928 02:42:18.567382 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:18.567385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:18.568426 (+ 1041us) server_negotiation.cc:407] Connection header received
0928 02:42:18.569726 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:18.569729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:18.569774 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:18.569847 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:18.570967 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:18.570972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:18.570975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:18.570988 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:18.571014 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:18.571031 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:18.571132 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:18.571289 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:42:18.571333 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:42:18.572557 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:18.567607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:36457 (local address 127.2.123.2:47779)
0928 02:42:18.568276 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:18.568318 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:18.569510 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:18.570021 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:18.570029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:18.570119 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:18.570681 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:18.570683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:18.570723 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:18.570803 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:18.571142 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:18.571152 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:18.571158 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:42:18.571316 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 02:42:18.571543 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260928 02:42:18.573119 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36457
--builtin_ntp_servers=127.2.123.20:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:42:18.573576 12601 heartbeater.cc:347] Connected to a master server at 127.2.123.62:36457
I20260928 02:42:18.573805 12601 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:18.574322 12601 heartbeater.cc:511] Master 127.2.123.62:36457 requested a full tablet report, sending...
I20260928 02:42:18.575481 12282 ts_manager.cc:194] Registered new tserver with Master: d4d4e06888c64b32a7e68b67f7b250b0 (127.2.123.2:33147)
W20260928 02:42:18.680511 12604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:18.680799 12604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:18.680852 12604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:18.685020 12604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:18.685084 12604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:18.685141 12604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:42:18.690115 12604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:18.691196 12604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:18.692574 12604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:18.702284 12612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:18.702397 12610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:18.702255 12609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:18.702538 12604 server_base.cc:1034] running on GCE node
I20260928 02:42:18.702899 12604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:18.703557 12604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:18.704721 12604 hybrid_clock.cc:648] HybridClock initialized: now 1790563338704702 us; error 31 us; skew 500 ppm
I20260928 02:42:18.706619 12604 webserver.cc:460] Webserver started at http://127.2.123.3:37267/ using document root <none> and password file <none>
I20260928 02:42:18.707180 12604 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:18.707242 12604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:18.707444 12604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:18.709523 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "1955d21a43b648ecadf6997d311da8b9"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "1b6f3a70002ceae15079d473bba278b2"
server_key_iv: "314dcc52df80b2fbcd929b6e0c511ee3"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.710103 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "1955d21a43b648ecadf6997d311da8b9"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "1b6f3a70002ceae15079d473bba278b2"
server_key_iv: "314dcc52df80b2fbcd929b6e0c511ee3"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.713780 12604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:42:18.716190 12618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.717401 12604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:18.717511 12604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "1955d21a43b648ecadf6997d311da8b9"
format_stamp: "Formatted at 2026-09-28 02:42:18 on dist-test-slave-rqvb"
server_key: "1b6f3a70002ceae15079d473bba278b2"
server_key_iv: "314dcc52df80b2fbcd929b6e0c511ee3"
server_key_version: "encryptionkey@0"
I20260928 02:42:18.717609 12604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:18.722206 12604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:18.722813 12604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:18.722957 12604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:18.723448 12604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:18.724289 12604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:18.724334 12604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.724380 12604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:18.724412 12604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:18.732415 12604 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:43343
I20260928 02:42:18.732460 12731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:43343 every 8 connection(s)
I20260928 02:42:18.733278 12604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:18.739398 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 12604
I20260928 02:42:18.739509 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:42:18.739583 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:18.735081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:46801 (local address 127.2.123.62:36457)
0928 02:42:18.735240 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:18.735244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:18.736307 (+ 1063us) server_negotiation.cc:407] Connection header received
0928 02:42:18.737568 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:18.737570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:18.737607 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:18.737689 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:18.739053 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:18.739057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:18.739060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:42:18.739072 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:18.739093 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:42:18.739101 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:18.739180 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:18.739396 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:42:18.739440 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:42:18.739794 2540 external_mini_cluster.cc:1428] Setting key 3145105a2a06c0cb7a53fe5991885298
I20260928 02:42:18.740520 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:18.735475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:36457 (local address 127.2.123.3:46801)
0928 02:42:18.736168 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:18.736207 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:18.737374 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:18.737894 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:18.737902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:18.738047 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:18.738695 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:42:18.738697 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:18.738749 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:42:18.738903 (+ 154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:18.739219 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:18.739227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:18.739234 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:42:18.739412 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 02:42:18.739655 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260928 02:42:18.742403 12732 heartbeater.cc:347] Connected to a master server at 127.2.123.62:36457
I20260928 02:42:18.742599 12732 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:18.742987 12732 heartbeater.cc:511] Master 127.2.123.62:36457 requested a full tablet report, sending...
I20260928 02:42:18.743880 12282 ts_manager.cc:194] Registered new tserver with Master: 1955d21a43b648ecadf6997d311da8b9 (127.2.123.3:43343)
I20260928 02:42:18.753916 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:42:18.756597 12321 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:42:18.755835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.2.123.62:36457)
0928 02:42:18.755961 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:18.755964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:18.756109 (+ 145us) server_negotiation.cc:407] Connection header received
0928 02:42:18.756205 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:18.756207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:18.756240 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:18.756316 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:18.756485 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47802: BlockingRecv error: recv got EOF from 127.0.0.1:47802 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:42:18.756764 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 12342
I20260928 02:42:18.761354 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 12473
I20260928 02:42:18.765667 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 12604
I20260928 02:42:18.769722 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 12251
2026-09-28T02:42:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (797 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:18 dist-test-slave-rqvb krb5kdc[12742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:18 dist-test-slave-rqvb krb5kdc[12742](info): set up 2 sockets
Sep 28 02:42:18 dist-test-slave-rqvb krb5kdc[12742](info): commencing operation
krb5kdc: starting...
W20260928 02:42:19.718704 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:42:19 dist-test-slave-rqvb krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:42:19.834976 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40753
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:19.945122 12755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:19.945396 12755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:19.945436 12755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:19.949386 12755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:42:19.949445 12755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:19.949462 12755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:19.949478 12755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:42:19.949492 12755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:42:19.954860 12755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:19.955865 12755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:19.956933 12755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:19.964993 12761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:19.965014 12763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:19.965101 12755 server_base.cc:1034] running on GCE node
W20260928 02:42:19.965107 12760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:19.965888 12755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:19.966954 12755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:19.968113 12755 hybrid_clock.cc:648] HybridClock initialized: now 1790563339968095 us; error 31 us; skew 500 ppm
Sep 28 02:42:19 dist-test-slave-rqvb krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:19.970765 12755 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:42:19.971731 12755 webserver.cc:460] Webserver started at http://127.2.123.62:34795/ using document root <none> and password file <none>
I20260928 02:42:19.972278 12755 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:19.972347 12755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:19.972553 12755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:19.974668 12755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "38bd2c767b824b9d92f29755157eaf12"
format_stamp: "Formatted at 2026-09-28 02:42:19 on dist-test-slave-rqvb"
server_key: "7167df67add46e6e85ab167fe332a0f7"
server_key_iv: "a0129c4c5dace88c694b8ac86bb4a273"
server_key_version: "encryptionkey@0"
I20260928 02:42:19.975243 12755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "38bd2c767b824b9d92f29755157eaf12"
format_stamp: "Formatted at 2026-09-28 02:42:19 on dist-test-slave-rqvb"
server_key: "7167df67add46e6e85ab167fe332a0f7"
server_key_iv: "a0129c4c5dace88c694b8ac86bb4a273"
server_key_version: "encryptionkey@0"
I20260928 02:42:19.978904 12755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:42:19.981371 12770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:19.982635 12755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:19.982749 12755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "38bd2c767b824b9d92f29755157eaf12"
format_stamp: "Formatted at 2026-09-28 02:42:19 on dist-test-slave-rqvb"
server_key: "7167df67add46e6e85ab167fe332a0f7"
server_key_iv: "a0129c4c5dace88c694b8ac86bb4a273"
server_key_version: "encryptionkey@0"
I20260928 02:42:19.982844 12755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:19.997731 12755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:20.000941 12755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:20.001111 12755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:20.008026 12755 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:40753
I20260928 02:42:20.008337 12822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:40753 every 8 connection(s)
I20260928 02:42:20.009021 12755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:42:20.011145 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 12755
I20260928 02:42:20.011253 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:42:20.011504 2540 external_mini_cluster.cc:1428] Setting key 5b4df54d87fe4444af813c55c9188add
I20260928 02:42:20.011916 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:20.018931 12823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12: Bootstrap starting.
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:42:20.021245 12823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:20.022161 12823 log.cc:824] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:20.024377 12823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12: No bootstrap required, opened a new log
I20260928 02:42:20.026469 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.012934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.2.123.62:40753)
0928 02:42:20.013349 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.013367 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.013410 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:42:20.014188 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.014213 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.014645 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.015031 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:20.015653 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.017081 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.017922 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.018608 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.018768 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.021224 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:20.021231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:20.021248 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:20.021281 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:20.023286 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.023766 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.023772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.023778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.023919 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.024175 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.024178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.024180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.024688 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:20.024884 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.025118 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.025421 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260928 02:42:20.028668 12823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd2c767b824b9d92f29755157eaf12" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40753 } }
I20260928 02:42:20.028937 12823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:20.028985 12823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38bd2c767b824b9d92f29755157eaf12, State: Initialized, Role: FOLLOWER
I20260928 02:42:20.029381 12823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd2c767b824b9d92f29755157eaf12" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40753 } }
I20260928 02:42:20.029503 12823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:42:20.029546 12823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:42:20.029606 12823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:20.030521 12823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd2c767b824b9d92f29755157eaf12" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40753 } }
I20260928 02:42:20.030817 12823 leader_election.cc:302] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38bd2c767b824b9d92f29755157eaf12; no voters:
I20260928 02:42:20.031129 12823 leader_election.cc:288] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:42:20.031282 12828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:20.031543 12828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [term 1 LEADER]: Becoming Leader. State: Replica: 38bd2c767b824b9d92f29755157eaf12, State: Running, Role: LEADER
I20260928 02:42:20.031826 12828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd2c767b824b9d92f29755157eaf12" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40753 } }
I20260928 02:42:20.032368 12823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:42:20.033447 12829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38bd2c767b824b9d92f29755157eaf12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd2c767b824b9d92f29755157eaf12" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40753 } } }
I20260928 02:42:20.033452 12830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38bd2c767b824b9d92f29755157eaf12. Latest consensus state: current_term: 1 leader_uuid: "38bd2c767b824b9d92f29755157eaf12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd2c767b824b9d92f29755157eaf12" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40753 } } }
I20260928 02:42:20.033577 12829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:20.033586 12830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12 [sys.catalog]: This master's current role is: LEADER
I20260928 02:42:20.033916 12837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:42:20.036542 12837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:42:20.041555 12837 catalog_manager.cc:1347] Generated new cluster ID: a3ecc95b441c46dba4613d12757d95ad
I20260928 02:42:20.041617 12837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:42:20.053674 12837 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:42:20.054618 12837 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:42:20.061426 12837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12: Generated new TSK 0
I20260928 02:42:20.062089 12837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:42:20.090539 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40753
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:20.201718 12849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:20.202020 12849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:20.202072 12849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:20.206148 12849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:20.206207 12849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:20.206266 12849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:42:20.211551 12849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:20.212550 12849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:20.213511 12849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:20.222666 12857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:20.222715 12855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:20.222836 12849 server_base.cc:1034] running on GCE node
W20260928 02:42:20.222751 12854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:20.223409 12849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:20.223994 12849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:20.225145 12849 hybrid_clock.cc:648] HybridClock initialized: now 1790563340225132 us; error 28 us; skew 500 ppm
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:20.227845 12849 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:42:20.228672 12849 webserver.cc:460] Webserver started at http://127.2.123.1:37631/ using document root <none> and password file <none>
I20260928 02:42:20.229192 12849 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:20.229254 12849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:20.229444 12849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:20.231477 12849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "b8fd98e871854b7f93f4a8e89a5a48b2"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "2df029828b0aa5f03d62fbac586406e8"
server_key_iv: "0e2a0c16c93eed379cb49d37e41d2e93"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.232007 12849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "b8fd98e871854b7f93f4a8e89a5a48b2"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "2df029828b0aa5f03d62fbac586406e8"
server_key_iv: "0e2a0c16c93eed379cb49d37e41d2e93"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.235707 12849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 02:42:20.238353 12864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.239529 12849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:20.239626 12849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "b8fd98e871854b7f93f4a8e89a5a48b2"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "2df029828b0aa5f03d62fbac586406e8"
server_key_iv: "0e2a0c16c93eed379cb49d37e41d2e93"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.239739 12849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:20.259482 12849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:20.262679 12849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:20.262876 12849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:20.263469 12849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:20.264289 12849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:20.264334 12849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.264389 12849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:20.264416 12849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.273061 12849 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:37325
I20260928 02:42:20.273092 12977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:37325 every 8 connection(s)
I20260928 02:42:20.273994 12849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:42:20.276631 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 12849
I20260928 02:42:20.276738 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:42:20.277016 2540 external_mini_cluster.cc:1428] Setting key 07da03a8a1208fda1748d186724e2cc2
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:42:20.290153 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.275909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:44761 (local address 127.2.123.62:40753)
0928 02:42:20.276027 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.276030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.276895 (+ 865us) server_negotiation.cc:407] Connection header received
0928 02:42:20.278441 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.278445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.278499 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.278553 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:20.280154 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.280710 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.282409 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.282710 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.282827 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.285952 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:20.285955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:20.285986 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:20.286020 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:20.287668 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.288259 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.288262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.288263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.288349 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.288738 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.288742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.288743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.288990 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:20.289188 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.289815 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.289942 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:42:20.291029 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.276244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40753 (local address 127.2.123.1:44761)
0928 02:42:20.276731 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.276784 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:20.278197 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:20.278974 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:20.278983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:20.279451 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:20.279958 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.279977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.280843 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.280847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.282253 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.282265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.282919 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.282923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.283150 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.283846 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:20.283869 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:20.285686 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:20.287822 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:20.287828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:20.287844 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:20.288114 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:20.288429 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:20.288431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:20.288433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:20.288617 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:20.289113 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:20.289120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:20.289530 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 02:42:20.289710 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 02:42:20.290089 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":112,"threads_started":1}
I20260928 02:42:20.292333 12978 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40753
I20260928 02:42:20.292582 12978 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:20.293077 12978 heartbeater.cc:511] Master 127.2.123.62:40753 requested a full tablet report, sending...
I20260928 02:42:20.294720 12787 ts_manager.cc:194] Registered new tserver with Master: b8fd98e871854b7f93f4a8e89a5a48b2 (127.2.123.1:37325)
I20260928 02:42:20.295848 12787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:44761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:42:20.305290 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40753
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:20.411860 12983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:20.412122 12983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:20.412180 12983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:20.416138 12983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:20.416198 12983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:20.416276 12983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:42:20.421379 12983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:20.422379 12983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:20.423347 12983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:20.432448 12988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:20.432485 12983 server_base.cc:1034] running on GCE node
W20260928 02:42:20.432441 12991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:20.432441 12989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:20.433089 12983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:20.433662 12983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:20.434821 12983 hybrid_clock.cc:648] HybridClock initialized: now 1790563340434802 us; error 34 us; skew 500 ppm
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:20.437444 12983 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:42:20.438318 12983 webserver.cc:460] Webserver started at http://127.2.123.2:41769/ using document root <none> and password file <none>
I20260928 02:42:20.438851 12983 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:20.438910 12983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:20.439092 12983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:20.441155 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "275b0d5b911d462c8a882be10e4914bf"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "7697c9b9dae4c87fda5aedc8f991f6a2"
server_key_iv: "ade34baf4e136ee130ada70485e3a6eb"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.441707 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "275b0d5b911d462c8a882be10e4914bf"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "7697c9b9dae4c87fda5aedc8f991f6a2"
server_key_iv: "ade34baf4e136ee130ada70485e3a6eb"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.445503 12983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:42:20.448019 12998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.449244 12983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:42:20.449343 12983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "275b0d5b911d462c8a882be10e4914bf"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "7697c9b9dae4c87fda5aedc8f991f6a2"
server_key_iv: "ade34baf4e136ee130ada70485e3a6eb"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.449445 12983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:20.473716 12983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:20.476824 12983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:20.476995 12983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:20.477587 12983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:20.478420 12983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:20.478466 12983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.478506 12983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:20.478535 12983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.486758 12983 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:45889
I20260928 02:42:20.486814 13111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:45889 every 8 connection(s)
I20260928 02:42:20.487634 12983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:42:20.491485 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 12983
I20260928 02:42:20.491611 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:42:20.491917 2540 external_mini_cluster.cc:1428] Setting key 5cbde393f0cee255f070c7e2d3bbdc88
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:42:20.502239 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.489363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:35587 (local address 127.2.123.62:40753)
0928 02:42:20.489519 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.489523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.490465 (+ 942us) server_negotiation.cc:407] Connection header received
0928 02:42:20.492100 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.492103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.492151 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.492217 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:20.493552 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.494131 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.495532 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.495861 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.495985 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.498827 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:20.498830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:20.498832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:20.498856 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:20.500143 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.500728 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.500731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.500733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.500812 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.501166 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.501169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.501170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.501356 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:20.501448 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.501931 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.502074 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:42:20.503123 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.489803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40753 (local address 127.2.123.2:35587)
0928 02:42:20.490307 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.490350 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:20.491857 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:20.492390 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:20.492397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:20.492815 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:20.493405 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.493425 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.494251 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.494255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.495395 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.495405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.496018 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.496021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.496230 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.496819 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:20.496835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:20.498637 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:20.500260 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:20.500266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:20.500282 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:20.500623 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:20.500905 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:20.500909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:20.500911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:20.501080 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:20.501455 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:20.501462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:20.501707 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 02:42:20.501877 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:42:20.502273 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:42:20.504412 13112 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40753
I20260928 02:42:20.504663 13112 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:20.505141 13112 heartbeater.cc:511] Master 127.2.123.62:40753 requested a full tablet report, sending...
I20260928 02:42:20.506234 12787 ts_manager.cc:194] Registered new tserver with Master: 275b0d5b911d462c8a882be10e4914bf (127.2.123.2:45889)
I20260928 02:42:20.506772 12787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:35587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:42:20.518848 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40753
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:42:20.624895 13117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:42:20.625146 13117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:42:20.625187 13117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:42:20.629323 13117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:42:20.629376 13117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:42:20.629441 13117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:42:20.634771 13117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:42:20.635807 13117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:42:20.636795 13117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:42:20.646055 13122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:20.646116 13123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:42:20.646217 13125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:42:20.646245 13117 server_base.cc:1034] running on GCE node
I20260928 02:42:20.646709 13117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:42:20.647330 13117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:42:20.648507 13117 hybrid_clock.cc:648] HybridClock initialized: now 1790563340648482 us; error 37 us; skew 500 ppm
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:42:20.651266 13117 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:42:20.652127 13117 webserver.cc:460] Webserver started at http://127.2.123.3:37821/ using document root <none> and password file <none>
I20260928 02:42:20.652676 13117 fs_manager.cc:359] Metadata directory not provided
I20260928 02:42:20.652745 13117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:42:20.652940 13117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:42:20.655128 13117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "a3da691009794cfc96f2da3e21157db7"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "8fa45219cb92aabd92582d760d3b3d4e"
server_key_iv: "220d5cca5a11ca086f8b0f591a3f07be"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.655823 13117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "a3da691009794cfc96f2da3e21157db7"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "8fa45219cb92aabd92582d760d3b3d4e"
server_key_iv: "220d5cca5a11ca086f8b0f591a3f07be"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.659881 13117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 02:42:20.662400 13132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.663663 13117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:42:20.663758 13117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "a3da691009794cfc96f2da3e21157db7"
format_stamp: "Formatted at 2026-09-28 02:42:20 on dist-test-slave-rqvb"
server_key: "8fa45219cb92aabd92582d760d3b3d4e"
server_key_iv: "220d5cca5a11ca086f8b0f591a3f07be"
server_key_version: "encryptionkey@0"
I20260928 02:42:20.663846 13117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:42:20.681699 13117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:42:20.684655 13117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:42:20.684823 13117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:42:20.685395 13117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:42:20.686265 13117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:42:20.686331 13117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.686384 13117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:42:20.686414 13117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:42:20.694720 13117 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:34657
I20260928 02:42:20.694763 13245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:34657 every 8 connection(s)
I20260928 02:42:20.695595 13117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:42:20.704720 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 13117
I20260928 02:42:20.704810 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:42:20.705068 2540 external_mini_cluster.cc:1428] Setting key a58e7833e1b88097b872075c27111764
I20260928 02:42:20.708716 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.697405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:41237 (local address 127.2.123.62:40753)
0928 02:42:20.697569 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.697574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.698416 (+ 842us) server_negotiation.cc:407] Connection header received
0928 02:42:20.699736 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.699738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.699790 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.699870 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:20.701098 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.701656 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.702824 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.703044 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.703131 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.705452 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:20.705455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:20.705457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:20.705481 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:20.706896 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.707379 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.707382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.707383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.707455 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.707727 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.707728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.707729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.707885 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:20.707955 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.708486 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.708596 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:42:20.709503 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.697800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40753 (local address 127.2.123.3:41237)
0928 02:42:20.698272 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.698313 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:20.699521 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:20.700065 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:20.700071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:20.700421 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:42:20.700933 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.700954 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.701775 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.701778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.702712 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.702721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.703174 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.703177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.703323 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.703858 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:42:20.703875 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:42:20.705296 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:42:20.707048 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:20.707054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:20.707068 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:20.707302 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:20.707534 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:42:20.707536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:42:20.707538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:42:20.707662 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:42:20.707995 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:42:20.708000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:42:20.708274 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 02:42:20.708448 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:42:20.708704 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
I20260928 02:42:20.710492 13246 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40753
I20260928 02:42:20.710719 13246 heartbeater.cc:464] Registering TS with master...
I20260928 02:42:20.711211 13246 heartbeater.cc:511] Master 127.2.123.62:40753 requested a full tablet report, sending...
I20260928 02:42:20.712109 12787 ts_manager.cc:194] Registered new tserver with Master: a3da691009794cfc96f2da3e21157db7 (127.2.123.3:34657)
I20260928 02:42:20.712630 12787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:41237
I20260928 02:42:20.718617 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:42:20.726729 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.720382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.2.123.62:40753)
0928 02:42:20.720481 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.720485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.720627 (+ 142us) server_negotiation.cc:407] Connection header received
0928 02:42:20.720716 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.720718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.720752 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.720815 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:42:20.721392 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.721905 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.722774 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.722981 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.723095 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.724279 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:42:20.724282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:42:20.724284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:42:20.724305 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:42:20.725388 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.725771 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.725774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.725775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.725843 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:42:20.726069 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:42:20.726071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:42:20.726072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:42:20.726270 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:42:20.726373 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.726524 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.726616 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:42:20.729384 12787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:42:20.732328 12787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:42:20.746902 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.741738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.2.123.1:37325)
0928 02:42:20.742179 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.742193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.742923 (+ 730us) server_negotiation.cc:407] Connection header received
0928 02:42:20.743123 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.743128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.743308 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.743442 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:20.743641 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.744658 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.745647 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.746248 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.746346 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.746499 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.746669 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.746758 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":74,"threads_started":1}
I20260928 02:42:20.746889 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.741320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:37325 (local address 127.0.0.1:54846)
0928 02:42:20.742824 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.742837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:20.742927 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:20.743440 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:20.743443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:20.743460 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:20.743553 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.743559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.744759 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.744762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.745548 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.745558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.746357 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.746361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.746482 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.746520 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:42:20.746622 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:42:20.746743 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"mutex_wait_us":19,"thread_start_us":96,"threads_started":1}
I20260928 02:42:20.747200 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.740839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.2.123.2:45889)
0928 02:42:20.742197 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.742207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.742226 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:42:20.742289 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.742294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.742458 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.742590 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:20.742809 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.743929 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.745614 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.746312 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.746405 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.746540 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.746969 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.747065 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":136,"threads_started":1}
I20260928 02:42:20.747285 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.740734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:34657 (local address 127.0.0.1:35020)
0928 02:42:20.741065 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.741117 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:20.741249 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:20.742470 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:20.742476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:20.742496 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:20.742625 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.742631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.743921 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.743924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.745485 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.745495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.746931 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.746934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.747013 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.747033 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:42:20.747076 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:42:20.747134 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":133,"thread_start_us":79,"threads_started":1}
I20260928 02:42:20.747277 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.740734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:45889 (local address 127.0.0.1:57788)
0928 02:42:20.741823 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.741837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:20.741923 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:20.742580 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:20.742583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:20.742597 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:20.742681 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.742687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.744049 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.744052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.745486 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.745495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.746602 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.746606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.746694 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.746717 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:42:20.746786 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:42:20.746855 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":87,"thread_start_us":108,"threads_started":1}
I20260928 02:42:20.747956 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.741550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.2.123.3:34657)
0928 02:42:20.741947 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.741981 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.742005 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:42:20.742083 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.742088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.742299 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.742449 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:20.742797 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.743758 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.746057 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.746817 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.747546 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.747674 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.747737 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.747822 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":109,"threads_started":1}
I20260928 02:42:20.750298 12912 tablet_service.cc:1467] Processing CreateTablet for tablet de5928274f984e4abc114e3b772797fe (DEFAULT_TABLE table=test-table [id=403ca92414ee489cb91ba7e0669dbba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:20.750562 13046 tablet_service.cc:1467] Processing CreateTablet for tablet de5928274f984e4abc114e3b772797fe (DEFAULT_TABLE table=test-table [id=403ca92414ee489cb91ba7e0669dbba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:20.751456 12912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5928274f984e4abc114e3b772797fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:20.751456 13046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5928274f984e4abc114e3b772797fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:20.752441 13180 tablet_service.cc:1467] Processing CreateTablet for tablet de5928274f984e4abc114e3b772797fe (DEFAULT_TABLE table=test-table [id=403ca92414ee489cb91ba7e0669dbba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:42:20.753261 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5928274f984e4abc114e3b772797fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:42:20.758603 13263 tablet_bootstrap.cc:492] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2: Bootstrap starting.
I20260928 02:42:20.759001 13264 tablet_bootstrap.cc:492] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf: Bootstrap starting.
I20260928 02:42:20.759675 13265 tablet_bootstrap.cc:492] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: Bootstrap starting.
I20260928 02:42:20.761266 13265 tablet_bootstrap.cc:654] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:20.761524 13264 tablet_bootstrap.cc:654] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:20.762372 13264 log.cc:824] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:20.762418 13265 log.cc:824] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:20.764190 13264 tablet_bootstrap.cc:492] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf: No bootstrap required, opened a new log
I20260928 02:42:20.764370 13264 ts_tablet_manager.cc:1397] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:42:20.764603 13263 tablet_bootstrap.cc:654] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2: Neither blocks nor log segments found. Creating new log.
I20260928 02:42:20.764631 13265 tablet_bootstrap.cc:492] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: No bootstrap required, opened a new log
I20260928 02:42:20.764789 13265 ts_tablet_manager.cc:1397] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:42:20.765784 13263 log.cc:824] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:42:20.767540 13264 raft_consensus.cc:348] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.767752 13264 raft_consensus.cc:374] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:20.767755 13263 tablet_bootstrap.cc:492] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2: No bootstrap required, opened a new log
I20260928 02:42:20.767822 13264 raft_consensus.cc:729] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275b0d5b911d462c8a882be10e4914bf, State: Initialized, Role: FOLLOWER
I20260928 02:42:20.767917 13263 ts_tablet_manager.cc:1397] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 02:42:20.768267 13264 consensus_queue.cc:260] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.768676 13265 raft_consensus.cc:348] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.768982 13265 raft_consensus.cc:374] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:20.769068 13265 raft_consensus.cc:729] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3da691009794cfc96f2da3e21157db7, State: Initialized, Role: FOLLOWER
I20260928 02:42:20.769260 13264 ts_tablet_manager.cc:1428] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:42:20.769615 13112 heartbeater.cc:503] Master 127.2.123.62:40753 was elected leader, sending a full tablet report...
I20260928 02:42:20.769551 13265 consensus_queue.cc:260] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.770218 13246 heartbeater.cc:503] Master 127.2.123.62:40753 was elected leader, sending a full tablet report...
I20260928 02:42:20.770542 13265 ts_tablet_manager.cc:1428] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:42:20.771538 13263 raft_consensus.cc:348] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.771766 13263 raft_consensus.cc:374] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:42:20.771840 13263 raft_consensus.cc:729] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8fd98e871854b7f93f4a8e89a5a48b2, State: Initialized, Role: FOLLOWER
I20260928 02:42:20.772249 13263 consensus_queue.cc:260] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.772847 12978 heartbeater.cc:503] Master 127.2.123.62:40753 was elected leader, sending a full tablet report...
I20260928 02:42:20.773113 13263 ts_tablet_manager.cc:1428] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 02:42:20.775131 12979 tablet.cc:2369] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:42:20.796901 13270 raft_consensus.cc:482] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:42:20.797135 13270 raft_consensus.cc:504] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.798290 13270 leader_election.cc:288] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 275b0d5b911d462c8a882be10e4914bf (127.2.123.2:45889), b8fd98e871854b7f93f4a8e89a5a48b2 (127.2.123.1:37325)
I20260928 02:42:20.802232 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.798658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:39253 (local address 127.2.123.2:45889)
0928 02:42:20.798783 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.798788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.798804 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:42:20.798929 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.798932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.798994 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.799078 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:20.799355 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.800013 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.801307 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.801698 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.801784 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.801844 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.802044 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.802112 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:42:20.802246 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.798515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:45889 (local address 127.2.123.3:39253)
0928 02:42:20.798712 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.798726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:20.798816 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:20.799107 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:20.799111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:20.799131 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:20.799254 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.799261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.800110 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.800113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.801190 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.801199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.801796 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.801799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.801900 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.801919 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:42:20.802025 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:42:20.802094 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 02:42:20.802289 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.798721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:42745 (local address 127.2.123.1:37325)
0928 02:42:20.798832 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.798836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.799079 (+ 243us) server_negotiation.cc:407] Connection header received
0928 02:42:20.799166 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.799168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.799215 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.799287 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:42:20.799583 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.800131 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.801364 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.801752 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.801840 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.801897 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.802102 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.802170 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:42:20.802474 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.798573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:37325 (local address 127.2.123.3:42745)
0928 02:42:20.798981 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.798994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:42:20.799089 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:42:20.799306 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:42:20.799309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:42:20.799324 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:42:20.799431 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.799437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.800236 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.800239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.801254 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:42:20.801263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.801860 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.801864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:42:20.801994 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.802016 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:42:20.802083 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:42:20.802149 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":118,"threads_started":1}
I20260928 02:42:20.802847 13066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5928274f984e4abc114e3b772797fe" candidate_uuid: "a3da691009794cfc96f2da3e21157db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275b0d5b911d462c8a882be10e4914bf" is_pre_election: true
I20260928 02:42:20.802978 12932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5928274f984e4abc114e3b772797fe" candidate_uuid: "a3da691009794cfc96f2da3e21157db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" is_pre_election: true
I20260928 02:42:20.803160 13066 raft_consensus.cc:2454] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3da691009794cfc96f2da3e21157db7 in term 0.
I20260928 02:42:20.803236 12932 raft_consensus.cc:2454] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3da691009794cfc96f2da3e21157db7 in term 0.
I20260928 02:42:20.803617 13136 leader_election.cc:302] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275b0d5b911d462c8a882be10e4914bf, a3da691009794cfc96f2da3e21157db7; no voters:
I20260928 02:42:20.803865 13270 raft_consensus.cc:2781] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:42:20.803941 13270 raft_consensus.cc:482] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:42:20.804060 13270 raft_consensus.cc:3037] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:20.805023 13270 raft_consensus.cc:504] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.805332 13270 leader_election.cc:288] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [CANDIDATE]: Term 1 election: Requested vote from peers 275b0d5b911d462c8a882be10e4914bf (127.2.123.2:45889), b8fd98e871854b7f93f4a8e89a5a48b2 (127.2.123.1:37325)
I20260928 02:42:20.805655 12932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5928274f984e4abc114e3b772797fe" candidate_uuid: "a3da691009794cfc96f2da3e21157db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2"
I20260928 02:42:20.805665 13066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5928274f984e4abc114e3b772797fe" candidate_uuid: "a3da691009794cfc96f2da3e21157db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275b0d5b911d462c8a882be10e4914bf"
I20260928 02:42:20.805754 12932 raft_consensus.cc:3037] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:20.805759 13066 raft_consensus.cc:3037] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:42:20.806730 13066 raft_consensus.cc:2454] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3da691009794cfc96f2da3e21157db7 in term 1.
I20260928 02:42:20.806748 12932 raft_consensus.cc:2454] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3da691009794cfc96f2da3e21157db7 in term 1.
I20260928 02:42:20.806974 13134 leader_election.cc:302] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3da691009794cfc96f2da3e21157db7, b8fd98e871854b7f93f4a8e89a5a48b2; no voters:
I20260928 02:42:20.807132 13270 raft_consensus.cc:2781] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:42:20.807354 13270 raft_consensus.cc:686] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 1 LEADER]: Becoming Leader. State: Replica: a3da691009794cfc96f2da3e21157db7, State: Running, Role: LEADER
I20260928 02:42:20.807685 13270 consensus_queue.cc:237] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } }
I20260928 02:42:20.810549 12787 catalog_manager.cc:5495] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3da691009794cfc96f2da3e21157db7 (127.2.123.3). New cstate: current_term: 1 leader_uuid: "a3da691009794cfc96f2da3e21157db7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3da691009794cfc96f2da3e21157db7" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 34657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 } health_report { overall_health: UNKNOWN } } }
I20260928 02:42:20.824013 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.820696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.2.123.3:34657)
0928 02:42:20.820824 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:42:20.820828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:42:20.820842 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:42:20.820914 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:42:20.820917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:42:20.820973 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:42:20.821071 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:42:20.821297 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.821802 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.822696 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:42:20.822901 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:42:20.822982 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:42:20.823259 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:42:20.823652 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:42:20.823751 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:42:20.823814 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:42:20.823868 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:42:20.829617 12932 raft_consensus.cc:1261] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2 [term 1 FOLLOWER]: Refusing update from remote peer a3da691009794cfc96f2da3e21157db7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:20.829617 13066 raft_consensus.cc:1261] T de5928274f984e4abc114e3b772797fe P 275b0d5b911d462c8a882be10e4914bf [term 1 FOLLOWER]: Refusing update from remote peer a3da691009794cfc96f2da3e21157db7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:42:20.830487 13273 consensus_queue.cc:1035] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "275b0d5b911d462c8a882be10e4914bf" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:20.830711 13270 consensus_queue.cc:1035] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8fd98e871854b7f93f4a8e89a5a48b2" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:42:20.837793 13276 mvcc.cc:205] Tried to move back new op lower bound from 7334147444031381504 to 7334147443950956544. Current Snapshot: MvccSnapshot[applied={T|T < 7334147444031381504}]
I20260928 02:42:20.840971 13277 mvcc.cc:205] Tried to move back new op lower bound from 7334147444031381504 to 7334147443950956544. Current Snapshot: MvccSnapshot[applied={T|T < 7334147444031381504}]
I20260928 02:42:20.846138 12786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:20.846288 12786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:42:20.848546 12786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38bd2c767b824b9d92f29755157eaf12: Sending DeleteTablet for 3 replicas of tablet de5928274f984e4abc114e3b772797fe
I20260928 02:42:20.849315 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet de5928274f984e4abc114e3b772797fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:20 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:35020
I20260928 02:42:20.849308 13046 tablet_service.cc:1514] Processing DeleteTablet for tablet de5928274f984e4abc114e3b772797fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:20 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:57788
I20260928 02:42:20.849402 12912 tablet_service.cc:1514] Processing DeleteTablet for tablet de5928274f984e4abc114e3b772797fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:42:20 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:54846
I20260928 02:42:20.849892 13285 tablet_replica.cc:330] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: stopping tablet replica
I20260928 02:42:20.850090 13287 tablet_replica.cc:330] T de5928274f984e4abc114e3b772797fe P b8fd98e871854b7f93f4a8e89a5a48b2: stopping tablet replica
I20260928 02:42:20.850462 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 12849
I20260928 02:42:20.851794 13285 raft_consensus.cc:2229] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:42:20.852178 13285 raft_consensus.cc:2258] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:42:20.853855 13285 ts_tablet_manager.cc:1905] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:42:20.855360 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 12983
W20260928 02:42:20.855320 12772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:37325 (error 108)
I20260928 02:42:20.857085 13285 ts_tablet_manager.cc:1918] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:42:20.857197 13285 log.cc:1196] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/de5928274f984e4abc114e3b772797fe
I20260928 02:42:20.857844 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.857575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:37325 (local address 127.0.0.1:54848)
0928 02:42:20.857691 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.857768 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:37325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:42:20.858081 12772 catalog_manager.cc:4584] TS b8fd98e871854b7f93f4a8e89a5a48b2 (127.2.123.1:37325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de5928274f984e4abc114e3b772797fe: Network error: Client connection negotiation failed: client connection to 127.2.123.1:37325: connect: Connection refused (error 111)
I20260928 02:42:20.862365 13285 ts_tablet_manager.cc:1939] T de5928274f984e4abc114e3b772797fe P a3da691009794cfc96f2da3e21157db7: Deleting consensus metadata
I20260928 02:42:20.862540 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:42:20.862295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:45889 (local address 127.0.0.1:57802)
0928 02:42:20.862403 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:42:20.862456 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.2:45889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:42:20.862869 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 13117
I20260928 02:42:20.867671 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 12755
2026-09-28T02:42:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2100 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[13291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[13291](info): set up 2 sockets
Sep 28 02:42:20 dist-test-slave-rqvb krb5kdc[13291](info): commencing operation
krb5kdc: starting...
W20260928 02:42:21.822829 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:42:21 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:42:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/rangeradmin_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/rangeradmin_127.2.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/rangerlookup_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/rangerlookup_127.2.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/HTTP_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/HTTP_127.2.123.20@KRBTEST.COM.keytab.
I20260928 02:42:21.987063 2540 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCSnLDv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/postgres -l logfile start
2026-09-28 02:42:24.398 UTC [13321] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:42:24.398 UTC [13321] LOG: listening on IPv4 address "127.2.123.20", port 49081
2026-09-28 02:42:24.401 UTC [13321] LOG: listening on Unix socket "/tmp/.s.PGSQL.49081"
2026-09-28 02:42:24.422 UTC [13324] LOG: database system was shut down at 2026-09-28 02:42:22 UTC
2026-09-28 02:42:24.425 UTC [13321] LOG: database system is ready to accept connections
I20260928 02:42:25.298723 2540 mini_postgres.cc:96] Postgres bound to 49081
2026-09-28 02:42:25.304 UTC [13332] FATAL: database "slave" does not exist
127.2.123.20:49081 - accepting connections
I20260928 02:42:25.305447 2540 mini_ranger.cc:161] Starting Ranger...
I20260928 02:42:25.320207 2540 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:42:25.797729 2540 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:42:25.797806 2540 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-admin
2026-09-28 02:42:26,083 [I] DB FLAVOR :POSTGRES
2026-09-28 02:42:26,083 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:42:26,083 [I] Checking connection..
2026-09-28 02:42:26,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:42:26,340 [I] Checking connection passed.
2026-09-28 02:42:26,341 [I] --------- Verifying version history table ---------
2026-09-28 02:42:26,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:42:26,567 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:42:26,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:42:26,824 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:42:26,824 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:42:26,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:42:27.078 UTC [13396] WARNING: there is no transaction in progress
2026-09-28 02:42:27,084 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:42:27,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:42:27,339 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:42:27,339 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:42:27,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:42:27,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:42:27,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rqvb.c.gcp-upstream.internal','N') ;"
2026-09-28 02:42:28,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:42:28,299 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:42:28,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:42:28,539 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:42:28,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:42:28,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:42:29,019 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:42:29,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:42:30.012 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.030 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.046 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.062 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.077 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.095 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.149 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.155 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.173 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.183 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.194 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.208 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.220 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.227 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.233 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30.239 UTC [13504] WARNING: there is no transaction in progress
2026-09-28 02:42:30,958 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:42:30,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:42:31,197 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:42:31,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:42:31,428 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:42:31,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:42:31,664 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:42:31,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:42:31,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:42:32,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rqvb.c.gcp-upstream.internal';"
2026-09-28 02:42:32,367 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:42:32,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:42:32,593 [I] DB_PATCHES have already been applied
I20260928 02:42:32.599023 2540 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCSnLDv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:42:32.599124 2540 mini_ranger.cc:193] Using host: 127.2.123.20
I20260928 02:42:32.602274 2540 mini_ranger.cc:227] Ranger admin URL: http://127.2.123.20:43265
Sep 28, 2026 2:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:42:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:42:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:42:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:42:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:42:33.601413 2540 mini_ranger.cc:160] Time spent starting Ranger: real 8.296s user 0.002s sys 0.007s
Sep 28, 2026 2:42:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:42:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:42:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:42:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:42:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:42:54.286118 2540 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/rangerkms_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/rangerkms_127.2.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.123.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.123.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.123.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/HTTP_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.123.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/HTTP_127.2.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:42:54 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563374, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:42:54.370010 2540 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:42:54.379225 2540 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:42:55.258983 2540 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:42:55.259075 2540 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms
I20260928 02:42:55.259174 2540 mini_ranger_kms.cc:227] Using postgres at 127.2.123.20:49081
2026-09-28 02:42:55,428 [I] DB FLAVOR :POSTGRES
2026-09-28 02:42:55,428 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:42:55,428 [I] Checking connection
2026-09-28 02:42:55,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:42:55,686 [I] connection success
2026-09-28 02:42:55,686 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:42:55,687 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:42:55,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:42:55,944 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:42:55,944 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:42:55,944 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:42:55,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:49081/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:42:56,212 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:42:56.650964 2540 mini_ranger_kms.cc:310] Created kms service
I20260928 02:42:56.846524 2540 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:42:56.886726 2540 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:42:56 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563374, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:42:57.234397 2540 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:42:57.234553 2540 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:42:57.234572 2540 mini_ranger_kms.cc:240] Using host: 127.2.123.20
I20260928 02:42:57.238389 2540 mini_ranger_kms.cc:276] Ranger KMS PID: 13684
I20260928 02:42:57.238538 2540 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.123.20:56115
Sep 28, 2026 2:42:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:42:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:42:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:42:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:42:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:42:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:42:59.341634 2540 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.972s user 0.021s sys 0.133s
I20260928 02:42:59.341773 2540 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:42:59.341904 2540 mini_ranger_kms.cc:401] 127.2.123.20:56115/kms/v1/keys
Sep 28 02:42:59 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.123.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:42:59 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.123.20@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
Sep 28 02:43:00 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.123.20@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
Sep 28 02:43:02 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:43:02.645252 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:36041
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:36041
--ranger_config_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:56115/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:02.758100 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:02.758407 13735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:02.758464 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:02.762550 13735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:02.762622 13735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:02.762648 13735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:02.762668 13735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:02.762689 13735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:02.768445 13735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.123.20:56115/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:36041
--ranger_config_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:02.769600 13735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:02.770720 13735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:02.779631 13741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:02.779608 13743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:02.779614 13735 server_base.cc:1034] running on GCE node
W20260928 02:43:02.779596 13740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:02.780488 13735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:02.781633 13735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:02.782836 13735 hybrid_clock.cc:648] HybridClock initialized: now 1790563382782801 us; error 50 us; skew 500 ppm
Sep 28 02:43:02 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:02.789891 13735 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:43:02.790943 13735 webserver.cc:460] Webserver started at http://127.2.123.62:36407/ using document root <none> and password file <none>
I20260928 02:43:02.791541 13735 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:02.791600 13735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:02.791793 13735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:43:02 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:02.847939 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "d96e659fab934547a9bcfa120ed43590"
format_stamp: "Formatted at 2026-09-28 02:43:02 on dist-test-slave-rqvb"
server_key: "b4b515507b515678250ce181691edadd"
server_key_iv: "597cb17058117766aece175c27e69ad6"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:02.848577 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "d96e659fab934547a9bcfa120ed43590"
format_stamp: "Formatted at 2026-09-28 02:43:02 on dist-test-slave-rqvb"
server_key: "b4b515507b515678250ce181691edadd"
server_key_iv: "597cb17058117766aece175c27e69ad6"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:02.852903 13735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260928 02:43:02.867322 13750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:02.868716 13735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:43:02.868852 13735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "d96e659fab934547a9bcfa120ed43590"
format_stamp: "Formatted at 2026-09-28 02:43:02 on dist-test-slave-rqvb"
server_key: "b4b515507b515678250ce181691edadd"
server_key_iv: "597cb17058117766aece175c27e69ad6"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:02.868996 13735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:02.883118 13735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:02.886085 13735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:02.886261 13735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:02.893584 13735 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:36041
I20260928 02:43:02.893600 13802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:36041 every 8 connection(s)
I20260928 02:43:02.894663 13735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:02.897493 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:02.903126 13803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590: Bootstrap starting.
I20260928 02:43:02.903204 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 13735
I20260928 02:43:02.903321 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:43:02.903614 2540 external_mini_cluster.cc:1428] Setting key 9e9f3f7a517b7c520f26cbab4334f0f7
I20260928 02:43:02.905879 13803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:02.906615 13803 log.cc:824] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:02.908710 13803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590: No bootstrap required, opened a new log
I20260928 02:43:02.912892 13803 raft_consensus.cc:348] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e659fab934547a9bcfa120ed43590" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36041 } }
I20260928 02:43:02.913192 13803 raft_consensus.cc:374] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:02.913237 13803 raft_consensus.cc:729] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96e659fab934547a9bcfa120ed43590, State: Initialized, Role: FOLLOWER
I20260928 02:43:02.913693 13803 consensus_queue.cc:260] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e659fab934547a9bcfa120ed43590" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36041 } }
Sep 28 02:43:02 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:02.913810 13803 raft_consensus.cc:388] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:02.913898 13803 raft_consensus.cc:482] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:02.914017 13803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:02.915095 13803 raft_consensus.cc:504] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e659fab934547a9bcfa120ed43590" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36041 } }
I20260928 02:43:02.915498 13803 leader_election.cc:302] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d96e659fab934547a9bcfa120ed43590; no voters:
I20260928 02:43:02.915804 13803 leader_election.cc:288] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:02.915902 13808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:02.916091 13808 raft_consensus.cc:686] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [term 1 LEADER]: Becoming Leader. State: Replica: d96e659fab934547a9bcfa120ed43590, State: Running, Role: LEADER
I20260928 02:43:02.916437 13808 consensus_queue.cc:237] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e659fab934547a9bcfa120ed43590" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36041 } }
I20260928 02:43:02.917126 13803 sys_catalog.cc:564] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:02.917609 13808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d96e659fab934547a9bcfa120ed43590. Latest consensus state: current_term: 1 leader_uuid: "d96e659fab934547a9bcfa120ed43590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e659fab934547a9bcfa120ed43590" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36041 } } }
I20260928 02:43:02.918283 13809 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d96e659fab934547a9bcfa120ed43590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e659fab934547a9bcfa120ed43590" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36041 } } }
I20260928 02:43:02.918401 13808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:02.918467 13809 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:02.918860 13817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:02.919184 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:02.905017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.2.123.62:36041)
0928 02:43:02.905502 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:02.905521 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:02.905566 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:43:02.906450 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:02.906476 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:02.906928 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:02.907291 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:02.908117 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:02.909661 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:02.910922 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:02.911413 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:02.911539 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:02.914398 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:02.914403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:02.914418 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:02.914446 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:02.916042 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:02.916555 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:02.916561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:02.916567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:02.916690 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:02.916941 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:02.916945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:02.916946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:02.917383 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:02.917557 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:02.917817 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:43:02.918138 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260928 02:43:02.922242 13817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:02.922659 13803 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:43:02.931795 13817 catalog_manager.cc:1347] Generated new cluster ID: fd327108851b4d49862e2f402179920c
I20260928 02:43:02.931887 13817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:02.942759 13817 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:02.944118 13817 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:02.950878 13817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590: Generated new TSK 0
I20260928 02:43:02.951814 13817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:43:03.680029 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:56115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36041
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:03.801713 13848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:03.802029 13848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:03.802119 13848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:03.806548 13848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:03.806682 13848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:03.806795 13848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:43:03.814215 13848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.123.20:56115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:03.815827 13848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:03.817075 13848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:03.827548 13855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:03.827668 13848 server_base.cc:1034] running on GCE node
W20260928 02:43:03.828032 13856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:03.828138 13858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:03.828675 13848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:03.829329 13848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:03.830489 13848 hybrid_clock.cc:648] HybridClock initialized: now 1790563383830466 us; error 34 us; skew 500 ppm
Sep 28 02:43:03 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:03.833204 13848 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:43:03.834218 13848 webserver.cc:460] Webserver started at http://127.2.123.1:46719/ using document root <none> and password file <none>
I20260928 02:43:03.834765 13848 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:03.834830 13848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:03.835036 13848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:43:03 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:03.854538 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "b160667d08424946bb37655b49c6fc97"
format_stamp: "Formatted at 2026-09-28 02:43:03 on dist-test-slave-rqvb"
server_key: "87797c4fb3a93ac8565e55997c667990"
server_key_iv: "e23b01a2f36911492243e83068ea484c"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:03.855418 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "b160667d08424946bb37655b49c6fc97"
format_stamp: "Formatted at 2026-09-28 02:43:03 on dist-test-slave-rqvb"
server_key: "87797c4fb3a93ac8565e55997c667990"
server_key_iv: "e23b01a2f36911492243e83068ea484c"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:03.863590 13848 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260928 02:43:03.889309 13865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:03.891196 13848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:43:03.891433 13848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "b160667d08424946bb37655b49c6fc97"
format_stamp: "Formatted at 2026-09-28 02:43:03 on dist-test-slave-rqvb"
server_key: "87797c4fb3a93ac8565e55997c667990"
server_key_iv: "e23b01a2f36911492243e83068ea484c"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:03.891672 13848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:03.914616 13848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:03.918640 13848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:03.918996 13848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:03.919806 13848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:03.920945 13848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:03.921053 13848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:03.921173 13848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:03.921249 13848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:03.932405 13848 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:43881
I20260928 02:43:03.932466 13979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:43881 every 8 connection(s)
I20260928 02:43:03.933736 13848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:03.938509 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 13848
I20260928 02:43:03.938637 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:03.938936 2540 external_mini_cluster.cc:1428] Setting key ad535665998310e27c747fb3564c53ba
Sep 28 02:43:03 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:03.952634 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:03.936464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:36041 (local address 127.2.123.1:35103)
0928 02:43:03.936964 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:03.937009 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:03.937870 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:03.938529 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:03.938539 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:03.938891 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:03.939594 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:03.939617 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:03.941475 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:03.941479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:03.942944 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:03.942957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:03.943638 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:03.943642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:03.943880 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:03.944590 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:03.944613 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:03.946441 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:03.949053 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:03.949060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:03.949079 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:03.949465 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:03.949821 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:03.949825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:03.949827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:03.950058 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:03.950545 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:03.950552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:03.950922 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 02:43:03.951120 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 02:43:03.951506 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":126,"threads_started":1}
I20260928 02:43:03.954095 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:03.935938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:35103 (local address 127.2.123.62:36041)
0928 02:43:03.936205 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:03.936211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:03.937119 (+ 908us) server_negotiation.cc:407] Connection header received
0928 02:43:03.938117 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:03.938121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:03.938181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:03.938278 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:03.939767 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:03.941301 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:03.943087 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:03.943434 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:03.946648 (+ 3214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:03.946801 (+ 153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:03.946816 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:03.946819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:03.946839 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:03.948913 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:03.949595 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:03.949598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:03.949600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:03.949695 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:03.950177 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:03.950180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:03.950182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:03.950423 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:03.953647 (+ 3224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:03.953732 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:43:03.953882 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260928 02:43:03.958212 13980 heartbeater.cc:347] Connected to a master server at 127.2.123.62:36041
I20260928 02:43:03.958519 13980 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:03.959175 13980 heartbeater.cc:511] Master 127.2.123.62:36041 requested a full tablet report, sending...
I20260928 02:43:03.961119 13760 ts_manager.cc:194] Registered new tserver with Master: b160667d08424946bb37655b49c6fc97 (127.2.123.1:43881)
I20260928 02:43:03.962731 13760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:35103
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:43:03.993281 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:56115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36041
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:43:04.150928 13987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:04.151353 13987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:04.151420 13987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:04.157639 13987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:04.157734 13987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:04.157823 13987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:43:04.165944 13987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.123.20:56115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:04.167431 13987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:04.168798 13987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:04.181257 13994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:04.181252 13996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:04.181821 13993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:04.183096 13987 server_base.cc:1034] running on GCE node
I20260928 02:43:04.183672 13987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:04.184556 13987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:04.187863 13987 hybrid_clock.cc:648] HybridClock initialized: now 1790563384187828 us; error 43 us; skew 500 ppm
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:04.192291 13987 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:43:04.193631 13987 webserver.cc:460] Webserver started at http://127.2.123.2:39329/ using document root <none> and password file <none>
I20260928 02:43:04.194448 13987 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:04.194545 13987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:04.194844 13987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:04.219241 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "2bcde1a298aa4dbb8e46808589290c4f"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-rqvb"
server_key: "626c867b794fd7b80afed2dfc305faf1"
server_key_iv: "10a3fc46ef643cd034586f00a145a69d"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:04.220095 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "2bcde1a298aa4dbb8e46808589290c4f"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-rqvb"
server_key: "626c867b794fd7b80afed2dfc305faf1"
server_key_iv: "10a3fc46ef643cd034586f00a145a69d"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:04.226301 13987 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 02:43:04.255163 14003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:04.257107 13987 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260928 02:43:04.257331 13987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "2bcde1a298aa4dbb8e46808589290c4f"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-rqvb"
server_key: "626c867b794fd7b80afed2dfc305faf1"
server_key_iv: "10a3fc46ef643cd034586f00a145a69d"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:04.257570 13987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:04.290512 13987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:04.294649 13987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:04.294842 13987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:04.295603 13987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:04.296752 13987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:04.296828 13987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:04.296942 13987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:04.296998 13987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:04.308161 13987 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:45959
I20260928 02:43:04.308211 14116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:45959 every 8 connection(s)
I20260928 02:43:04.309463 13987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:04.313822 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 13987
I20260928 02:43:04.313952 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:04.314298 2540 external_mini_cluster.cc:1428] Setting key 4846ac515365fd9220d4f8f5e92fd0db
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:04.351578 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.311883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:55539 (local address 127.2.123.62:36041)
0928 02:43:04.312013 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.312016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.313367 (+ 1351us) server_negotiation.cc:407] Connection header received
0928 02:43:04.314830 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.314836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.314906 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.314979 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:04.316915 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.317692 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.319533 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.319894 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.319977 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.347513 (+ 27536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:04.347518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:04.347520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:04.347557 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:04.349052 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:04.349677 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:04.349681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:04.349683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:04.349820 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:04.350253 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:04.350256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:04.350258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:04.350455 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:04.350540 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.351205 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.351343 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:43:04.352586 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.312382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:36041 (local address 127.2.123.2:55539)
0928 02:43:04.313169 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:04.313236 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:04.314516 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:04.315501 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:04.315512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:04.315918 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:04.316701 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.316737 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.317934 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.317938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.319401 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.319412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.320195 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.320199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.320409 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.321029 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:04.321051 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:04.347160 (+ 26109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:04.349230 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:04.349236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:04.349252 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:04.349549 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:04.349920 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:04.349924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:04.349927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:04.350150 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:04.350559 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:04.350568 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:04.350930 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 02:43:04.351149 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 02:43:04.351509 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":161,"threads_started":1}
I20260928 02:43:04.353129 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:56115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36041
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:04.354182 14117 heartbeater.cc:347] Connected to a master server at 127.2.123.62:36041
I20260928 02:43:04.354441 14117 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:04.355034 14117 heartbeater.cc:511] Master 127.2.123.62:36041 requested a full tablet report, sending...
I20260928 02:43:04.356357 13760 ts_manager.cc:194] Registered new tserver with Master: 2bcde1a298aa4dbb8e46808589290c4f (127.2.123.2:45959)
I20260928 02:43:04.357049 13760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:55539
W20260928 02:43:04.515245 14122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:04.516121 14122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:04.516274 14122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:04.522575 14122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:04.522734 14122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:04.522867 14122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:43:04.530759 14122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.123.20:56115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:04.532354 14122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:04.533725 14122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:04.546820 14128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:04.547211 14122 server_base.cc:1034] running on GCE node
W20260928 02:43:04.547511 14127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:04.550004 14130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:04.550725 14122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:04.551574 14122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:04.558957 14122 hybrid_clock.cc:648] HybridClock initialized: now 1790563384558864 us; error 97 us; skew 500 ppm
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:04.562999 14122 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:43:04.564177 14122 webserver.cc:460] Webserver started at http://127.2.123.3:35417/ using document root <none> and password file <none>
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:04.565533 14122 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:04.565619 14122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:04.565860 14122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:04.597909 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "90be0c45faa7457798fcb588c6603724"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-rqvb"
server_key: "cb8b5e2e04e91dd9a9750bc237a1c0eb"
server_key_iv: "6712431b22b952c82d8bbbdf1ede25ef"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:04.598748 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "90be0c45faa7457798fcb588c6603724"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-rqvb"
server_key: "cb8b5e2e04e91dd9a9750bc237a1c0eb"
server_key_iv: "6712431b22b952c82d8bbbdf1ede25ef"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:04.617833 14122 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.005s sys 0.000s
I20260928 02:43:04.646432 14137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:04.648219 14122 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 02:43:04.648391 14122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "90be0c45faa7457798fcb588c6603724"
format_stamp: "Formatted at 2026-09-28 02:43:04 on dist-test-slave-rqvb"
server_key: "cb8b5e2e04e91dd9a9750bc237a1c0eb"
server_key_iv: "6712431b22b952c82d8bbbdf1ede25ef"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:04.648548 14122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:04.671312 14122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:04.675123 14122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:04.675406 14122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:04.676216 14122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:04.677479 14122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:04.677695 14122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:04.677795 14122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:04.677856 14122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:04.690877 14122 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:38641
I20260928 02:43:04.692127 14122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:04.692377 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 14122
I20260928 02:43:04.692492 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:04.692770 2540 external_mini_cluster.cc:1428] Setting key e1a174042ec337f3835f21e81d8beac1
I20260928 02:43:04.696794 14250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:38641 every 8 connection(s)
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:04.716325 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.700773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:56615 (local address 127.2.123.62:36041)
0928 02:43:04.700896 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.700901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.701347 (+ 446us) server_negotiation.cc:407] Connection header received
0928 02:43:04.702653 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.702657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.702714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.702768 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:04.704508 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.705422 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.708818 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.709164 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.709238 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.712188 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:04.712192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:04.712195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:04.712223 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:04.713844 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:04.714488 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:04.714491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:04.714493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:04.714584 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:04.715016 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:04.715019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:04.715022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:04.715245 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:04.715324 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.716030 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.716157 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:43:04.720358 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.699657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:36041 (local address 127.2.123.3:56615)
0928 02:43:04.701188 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:04.701247 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:04.702388 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:04.703205 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:04.703215 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:04.703607 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:04.704338 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.704360 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.707219 (+ 2859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.707223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.708684 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.708695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.709433 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.709437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.709620 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.710288 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:04.710308 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:04.712005 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:04.714009 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:04.714015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:04.714033 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:04.714374 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:04.714708 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:04.714711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:04.714713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:04.714909 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:04.715390 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:04.715397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:04.715753 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 02:43:04.716818 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0928 02:43:04.719038 (+ 2220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"thread_start_us":142,"threads_started":1}
I20260928 02:43:04.722020 14251 heartbeater.cc:347] Connected to a master server at 127.2.123.62:36041
I20260928 02:43:04.722236 14251 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:04.722942 14251 heartbeater.cc:511] Master 127.2.123.62:36041 requested a full tablet report, sending...
I20260928 02:43:04.724258 13760 ts_manager.cc:194] Registered new tserver with Master: 90be0c45faa7457798fcb588c6603724 (127.2.123.3:38641)
I20260928 02:43:04.724982 13760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:56615
I20260928 02:43:04.730579 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:43:04.742394 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.732906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.2.123.62:36041)
0928 02:43:04.733013 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.733016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.733617 (+ 601us) server_negotiation.cc:407] Connection header received
0928 02:43:04.733818 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.733821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.733870 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.733920 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:04.734969 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.735776 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.736922 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.737237 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.737304 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.739019 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:04.739022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:04.739025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:04.739050 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:04.740576 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:04.741038 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:04.741042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:04.741043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:04.741120 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:04.741379 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:04.741382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:04.741384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:04.741589 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:04.741693 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.742107 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.742228 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:04.746063 13760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:04.750406 13760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:04.779507 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.764606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:45959 (local address 127.0.0.1:42236)
0928 02:43:04.766812 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:04.766826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:04.766905 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:04.773697 (+ 6792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:04.773700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:04.773718 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:04.773809 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.773853 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.775499 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.775502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.777621 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.777631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.778962 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.778966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.779088 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.779136 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:43:04.779222 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:43:04.779343 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2134,"thread_start_us":88,"threads_started":1}
I20260928 02:43:04.780596 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.764985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.2.123.2:45959)
0928 02:43:04.769878 (+ 4893us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.769894 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.769918 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:43:04.770013 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.770019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.770221 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.774062 (+ 3841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:04.774206 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.775368 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.777780 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.778835 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.780140 (+ 1305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.780302 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.780356 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.780454 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4785,"thread_start_us":88,"threads_started":1}
I20260928 02:43:04.784864 14033 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7d325387474397b556a06add4d7443 (DEFAULT_TABLE table=test-table [id=31638446d33e4b0e9d9175e724f46045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:04.785416 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.762085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:38641 (local address 127.0.0.1:38330)
0928 02:43:04.766312 (+ 4227us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:04.766374 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:04.766517 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:04.768263 (+ 1746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:04.768289 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:04.768311 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:04.768435 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.768441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.771295 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.771298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.773537 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.773548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.784974 (+ 11426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.784978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.785076 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.785101 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:43:04.785186 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:43:04.785268 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4024,"thread_start_us":93,"threads_started":1}
I20260928 02:43:04.785866 14265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.763150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.2.123.3:38641)
0928 02:43:04.767626 (+ 4476us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.767639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.767663 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:43:04.767734 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.767739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.767954 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.768099 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:04.768557 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.771169 (+ 2612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.781243 (+ 10074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.784652 (+ 3409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.784759 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.784907 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.785608 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.785719 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4359,"thread_start_us":115,"threads_started":1}
I20260928 02:43:04.787403 14033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7d325387474397b556a06add4d7443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:04.789022 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7d325387474397b556a06add4d7443 (DEFAULT_TABLE table=test-table [id=31638446d33e4b0e9d9175e724f46045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:04.790127 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7d325387474397b556a06add4d7443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:04.798003 14271 tablet_bootstrap.cc:492] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Bootstrap starting.
I20260928 02:43:04.800731 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.765647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:43881 (local address 127.0.0.1:35190)
0928 02:43:04.767291 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:04.767304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:04.767378 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:04.782755 (+ 15377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:04.782758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:04.782780 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:04.782873 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.782880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.797681 (+ 14801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.797685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.799058 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.799073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.800296 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.800300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.800403 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.800434 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:43:04.800488 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:43:04.800576 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1593,"thread_start_us":87,"threads_started":1}
I20260928 02:43:04.800819 14271 tablet_bootstrap.cc:654] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:04.801717 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.766056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35190 (local address 127.2.123.1:43881)
0928 02:43:04.782219 (+ 16163us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.782226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.782254 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:43:04.782335 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.782341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.782574 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.783038 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:04.783195 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.784318 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.799249 (+ 14931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.800131 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.801273 (+ 1142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.801430 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.801482 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.801573 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16024,"thread_start_us":78,"threads_started":1}
I20260928 02:43:04.805540 13913 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7d325387474397b556a06add4d7443 (DEFAULT_TABLE table=test-table [id=31638446d33e4b0e9d9175e724f46045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:04.806679 13913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7d325387474397b556a06add4d7443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:04.810729 14271 log.cc:824] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:04.811619 14270 tablet_bootstrap.cc:492] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f: Bootstrap starting.
I20260928 02:43:04.815887 14270 tablet_bootstrap.cc:654] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:04.816349 14273 tablet_bootstrap.cc:492] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97: Bootstrap starting.
I20260928 02:43:04.816787 14270 log.cc:824] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:04.820653 14271 tablet_bootstrap.cc:492] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: No bootstrap required, opened a new log
I20260928 02:43:04.821031 14271 ts_tablet_manager.cc:1397] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260928 02:43:04.825876 14273 tablet_bootstrap.cc:654] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:04.829197 14271 raft_consensus.cc:348] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.832139 14273 log.cc:824] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:04.832322 14271 raft_consensus.cc:374] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:04.832428 14271 raft_consensus.cc:729] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90be0c45faa7457798fcb588c6603724, State: Initialized, Role: FOLLOWER
I20260928 02:43:04.833400 14271 consensus_queue.cc:260] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.833504 14270 tablet_bootstrap.cc:492] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f: No bootstrap required, opened a new log
I20260928 02:43:04.833726 14270 ts_tablet_manager.cc:1397] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.005s
I20260928 02:43:04.834664 14271 ts_tablet_manager.cc:1428] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260928 02:43:04.835078 14251 heartbeater.cc:503] Master 127.2.123.62:36041 was elected leader, sending a full tablet report...
I20260928 02:43:04.837554 14270 raft_consensus.cc:348] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.837854 14270 raft_consensus.cc:374] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:04.837916 14270 raft_consensus.cc:729] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcde1a298aa4dbb8e46808589290c4f, State: Initialized, Role: FOLLOWER
I20260928 02:43:04.838450 14270 consensus_queue.cc:260] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.839810 14270 ts_tablet_manager.cc:1428] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:43:04.840142 14117 heartbeater.cc:503] Master 127.2.123.62:36041 was elected leader, sending a full tablet report...
I20260928 02:43:04.841081 14273 tablet_bootstrap.cc:492] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97: No bootstrap required, opened a new log
I20260928 02:43:04.842242 14273 ts_tablet_manager.cc:1397] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.005s
I20260928 02:43:04.847033 14273 raft_consensus.cc:348] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.848333 14273 raft_consensus.cc:374] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:04.848469 14273 raft_consensus.cc:729] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b160667d08424946bb37655b49c6fc97, State: Initialized, Role: FOLLOWER
I20260928 02:43:04.849079 14273 consensus_queue.cc:260] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.850342 14273 ts_tablet_manager.cc:1428] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260928 02:43:04.850754 13980 heartbeater.cc:503] Master 127.2.123.62:36041 was elected leader, sending a full tablet report...
I20260928 02:43:04.907560 14278 raft_consensus.cc:482] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:04.907795 14278 raft_consensus.cc:504] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.909224 14278 leader_election.cc:288] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90be0c45faa7457798fcb588c6603724 (127.2.123.3:38641), 2bcde1a298aa4dbb8e46808589290c4f (127.2.123.2:45959)
I20260928 02:43:04.920140 14265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.910497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:48159 (local address 127.2.123.3:38641)
0928 02:43:04.910642 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.910647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.911002 (+ 355us) server_negotiation.cc:407] Connection header received
0928 02:43:04.911228 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.911232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.911279 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.911325 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:04.912004 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.912827 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.918758 (+ 5931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.919411 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.919494 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.919565 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.919905 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.919987 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:43:04.921005 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.909611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:38641 (local address 127.2.123.1:48159)
0928 02:43:04.910908 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:04.910923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:04.911096 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:04.911468 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:04.911471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:04.911487 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:04.911625 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.911632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.912995 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.912998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.918613 (+ 5615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.918624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.919660 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.919663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.919767 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.919791 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:43:04.920772 (+ 981us) client_negotiation.cc:240] Negotiation successful
0928 02:43:04.920847 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"thread_start_us":117,"threads_started":1}
I20260928 02:43:04.923022 14189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7d325387474397b556a06add4d7443" candidate_uuid: "b160667d08424946bb37655b49c6fc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90be0c45faa7457798fcb588c6603724" is_pre_election: true
I20260928 02:43:04.923395 14189 raft_consensus.cc:2454] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b160667d08424946bb37655b49c6fc97 in term 0.
I20260928 02:43:04.924055 13866 leader_election.cc:302] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90be0c45faa7457798fcb588c6603724, b160667d08424946bb37655b49c6fc97; no voters:
I20260928 02:43:04.924388 14278 raft_consensus.cc:2781] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:04.924504 14278 raft_consensus.cc:482] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:04.924547 14278 raft_consensus.cc:3037] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:04.924789 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.910635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:56391 (local address 127.2.123.2:45959)
0928 02:43:04.910750 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.910754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.911933 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 02:43:04.914212 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.914216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.914271 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.914317 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:04.914707 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.915575 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.921292 (+ 5717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.921981 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.922088 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.922148 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.924569 (+ 2421us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.924666 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:04.925209 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.910050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:45959 (local address 127.2.123.1:56391)
0928 02:43:04.911856 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:04.911868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:04.914056 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:04.914445 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:04.914448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:04.914464 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:04.914581 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.914587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.915714 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.915717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.920673 (+ 4956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:04.920683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.922236 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.922240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:04.922341 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.922361 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:43:04.924975 (+ 2614us) client_negotiation.cc:240] Negotiation successful
0928 02:43:04.925057 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"thread_start_us":95,"threads_started":1}
I20260928 02:43:04.925827 14067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7d325387474397b556a06add4d7443" candidate_uuid: "b160667d08424946bb37655b49c6fc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bcde1a298aa4dbb8e46808589290c4f" is_pre_election: true
I20260928 02:43:04.925835 14278 raft_consensus.cc:504] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.926141 14067 raft_consensus.cc:2454] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b160667d08424946bb37655b49c6fc97 in term 0.
I20260928 02:43:04.926234 14278 leader_election.cc:288] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [CANDIDATE]: Term 1 election: Requested vote from peers 90be0c45faa7457798fcb588c6603724 (127.2.123.3:38641), 2bcde1a298aa4dbb8e46808589290c4f (127.2.123.2:45959)
I20260928 02:43:04.926712 14067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7d325387474397b556a06add4d7443" candidate_uuid: "b160667d08424946bb37655b49c6fc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bcde1a298aa4dbb8e46808589290c4f"
I20260928 02:43:04.926816 14067 raft_consensus.cc:3037] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:04.926961 14189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7d325387474397b556a06add4d7443" candidate_uuid: "b160667d08424946bb37655b49c6fc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90be0c45faa7457798fcb588c6603724"
I20260928 02:43:04.927070 14189 raft_consensus.cc:3037] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:04.928030 14067 raft_consensus.cc:2454] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b160667d08424946bb37655b49c6fc97 in term 1.
I20260928 02:43:04.928264 14189 raft_consensus.cc:2454] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b160667d08424946bb37655b49c6fc97 in term 1.
I20260928 02:43:04.928509 13868 leader_election.cc:302] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcde1a298aa4dbb8e46808589290c4f, b160667d08424946bb37655b49c6fc97; no voters:
I20260928 02:43:04.928753 14278 raft_consensus.cc:2781] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:04.928953 14278 raft_consensus.cc:686] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [term 1 LEADER]: Becoming Leader. State: Replica: b160667d08424946bb37655b49c6fc97, State: Running, Role: LEADER
I20260928 02:43:04.929314 14278 consensus_queue.cc:237] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } }
I20260928 02:43:04.932920 13767 catalog_manager.cc:5495] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 reported cstate change: term changed from 0 to 1, leader changed from <none> to b160667d08424946bb37655b49c6fc97 (127.2.123.1). New cstate: current_term: 1 leader_uuid: "b160667d08424946bb37655b49c6fc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b160667d08424946bb37655b49c6fc97" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43881 } health_report { overall_health: HEALTHY } } }
W20260928 02:43:04.940033 13981 tablet.cc:2369] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:43:04.948189 14252 tablet.cc:2369] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:43:04 dist-test-slave-rqvb krb5kdc[13291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:04.966152 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:04.961030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.2.123.1:43881)
0928 02:43:04.961152 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:04.961157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:04.961217 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:43:04.961431 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:04.961435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:04.961507 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:04.961638 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:04.962029 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.962877 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.964313 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:04.964661 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:04.964754 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:04.965173 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:04.965746 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:04.965862 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:04.965924 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:43:04.966005 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:43:04.973325 14189 raft_consensus.cc:1261] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 1 FOLLOWER]: Refusing update from remote peer b160667d08424946bb37655b49c6fc97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:04.973325 14067 raft_consensus.cc:1261] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 1 FOLLOWER]: Refusing update from remote peer b160667d08424946bb37655b49c6fc97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:04.974129 14278 consensus_queue.cc:1035] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90be0c45faa7457798fcb588c6603724" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:04.974359 14281 consensus_queue.cc:1035] T 6f7d325387474397b556a06add4d7443 P b160667d08424946bb37655b49c6fc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bcde1a298aa4dbb8e46808589290c4f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 45959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:04.986580 14287 mvcc.cc:205] Tried to move back new op lower bound from 7334147624843055104 to 7334147624674590720. Current Snapshot: MvccSnapshot[applied={T|T < 7334147624843055104}]
I20260928 02:43:04.990538 14285 mvcc.cc:205] Tried to move back new op lower bound from 7334147624843055104 to 7334147624674590720. Current Snapshot: MvccSnapshot[applied={T|T < 7334147624843055104 or (T in {7334147624843055104})}]
I20260928 02:43:04.995272 13760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:04.995463 13760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:04.997866 13760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d96e659fab934547a9bcfa120ed43590: Sending DeleteTablet for 3 replicas of tablet 6f7d325387474397b556a06add4d7443
I20260928 02:43:04.998683 14033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7d325387474397b556a06add4d7443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:04 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:42236
I20260928 02:43:04.998679 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7d325387474397b556a06add4d7443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:04 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:38330
I20260928 02:43:04.999202 14295 tablet_replica.cc:330] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: stopping tablet replica
I20260928 02:43:04.999425 13913 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7d325387474397b556a06add4d7443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:04 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:35190
I20260928 02:43:04.999581 14295 raft_consensus.cc:2229] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:04.999622 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 13848
I20260928 02:43:04.999877 14295 raft_consensus.cc:2258] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:05.001456 14295 ts_tablet_manager.cc:1905] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:05.004410 14295 ts_tablet_manager.cc:1918] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:05.004515 14295 log.cc:1196] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/6f7d325387474397b556a06add4d7443
I20260928 02:43:05.004840 14295 ts_tablet_manager.cc:1939] T 6f7d325387474397b556a06add4d7443 P 90be0c45faa7457798fcb588c6603724: Deleting consensus metadata
I20260928 02:43:05.005234 14294 tablet_replica.cc:330] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f: stopping tablet replica
I20260928 02:43:05.005923 14294 raft_consensus.cc:2229] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:05.006374 14294 raft_consensus.cc:2258] T 6f7d325387474397b556a06add4d7443 P 2bcde1a298aa4dbb8e46808589290c4f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:43:05.007117 13754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:43881 (error 108)
I20260928 02:43:05.007928 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 13987
I20260928 02:43:05.013201 13751 catalog_manager.cc:4841] TS 90be0c45faa7457798fcb588c6603724 (127.2.123.3:38641): tablet 6f7d325387474397b556a06add4d7443 (table test-table [id=31638446d33e4b0e9d9175e724f46045]) successfully deleted
I20260928 02:43:05.014420 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 14122
I20260928 02:43:05.016067 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:05.015758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:43881 (local address 127.0.0.1:35218)
0928 02:43:05.015879 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:05.015967 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:43881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:43:05.016220 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:05.015986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:45959 (local address 127.0.0.1:42244)
0928 02:43:05.016126 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:05.016169 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.2:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260928 02:43:05.016379 13753 catalog_manager.cc:4584] TS 2bcde1a298aa4dbb8e46808589290c4f (127.2.123.2:45959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f7d325387474397b556a06add4d7443: Network error: Client connection negotiation failed: client connection to 127.2.123.2:45959: connect: Connection refused (error 111)
I20260928 02:43:05.023167 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 13735
I20260928 02:43:05.032611 2540 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:43:05.367695 2540 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:43:05.367815 2540 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:43:05.703070 2540 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:43:05.703 UTC [13321] LOG: received smart shutdown request
2026-09-28 02:43:05.706 UTC [13321] LOG: background worker "logical replication launcher" (PID 13330) exited with exit code 1
2026-09-28 02:43:05.707 UTC [13325] LOG: shutting down
2026-09-28 02:43:05.872 UTC [13321] LOG: database system is shut down
2026-09-28T02:43:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45094 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:06 dist-test-slave-rqvb krb5kdc[14306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:06 dist-test-slave-rqvb krb5kdc[14306](info): set up 2 sockets
Sep 28 02:43:06 dist-test-slave-rqvb krb5kdc[14306](info): commencing operation
krb5kdc: starting...
W20260928 02:43:06.937896 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:43:07 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/rangeradmin_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/rangeradmin_127.2.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/rangerlookup_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/rangerlookup_127.2.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/HTTP_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/HTTP_127.2.123.20@KRBTEST.COM.keytab.
I20260928 02:43:07.104279 2540 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCSnLDv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/postgres -l logfile start
2026-09-28 02:43:09.597 UTC [14336] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:43:09.597 UTC [14336] LOG: listening on IPv4 address "127.2.123.20", port 39231
2026-09-28 02:43:09.601 UTC [14336] LOG: listening on Unix socket "/tmp/.s.PGSQL.39231"
2026-09-28 02:43:09.621 UTC [14339] LOG: database system was shut down at 2026-09-28 02:43:08 UTC
2026-09-28 02:43:09.624 UTC [14336] LOG: database system is ready to accept connections
I20260928 02:43:10.497567 2540 mini_postgres.cc:96] Postgres bound to 39231
2026-09-28 02:43:10.503 UTC [14347] FATAL: database "slave" does not exist
127.2.123.20:39231 - accepting connections
I20260928 02:43:10.504094 2540 mini_ranger.cc:161] Starting Ranger...
I20260928 02:43:10.518415 2540 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:43:10.990257 2540 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:43:10.990326 2540 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-admin
2026-09-28 02:43:11,260 [I] DB FLAVOR :POSTGRES
2026-09-28 02:43:11,260 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:43:11,260 [I] Checking connection..
2026-09-28 02:43:11,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:43:11,494 [I] Checking connection passed.
2026-09-28 02:43:11,495 [I] --------- Verifying version history table ---------
2026-09-28 02:43:11,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:43:11,752 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:43:11,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:43:12,008 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:43:12,008 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:43:12,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:43:12.268 UTC [14411] WARNING: there is no transaction in progress
2026-09-28 02:43:12,274 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:43:12,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:43:12,516 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:43:12,516 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:43:12,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:43:12,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:43:13,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rqvb.c.gcp-upstream.internal','N') ;"
2026-09-28 02:43:13,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:43:13,497 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:43:13,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:43:13,760 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:43:13,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:43:14,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:43:14,255 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:43:14,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:43:15.195 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.211 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.228 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.244 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.259 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.280 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.337 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.344 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.358 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.369 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.380 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.392 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.402 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.408 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.414 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:15.421 UTC [14519] WARNING: there is no transaction in progress
2026-09-28 02:43:16,039 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:43:16,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:43:16,273 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:43:16,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:43:16,521 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:43:16,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:43:16,769 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:43:16,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:43:17,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:43:17,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rqvb.c.gcp-upstream.internal';"
2026-09-28 02:43:17,494 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:43:17,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:43:17,739 [I] DB_PATCHES have already been applied
I20260928 02:43:17.744640 2540 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCSnLDv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:43:17.744705 2540 mini_ranger.cc:193] Using host: 127.2.123.20
I20260928 02:43:17.747951 2540 mini_ranger.cc:227] Ranger admin URL: http://127.2.123.20:36413
Sep 28, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:43:18.750634 2540 mini_ranger.cc:160] Time spent starting Ranger: real 8.247s user 0.004s sys 0.004s
Sep 28, 2026 2:43:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:43:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:43:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:43:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:43:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:43:40.627609 2540 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/rangerkms_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/rangerkms_127.2.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.123.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.123.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.123.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/HTTP_127.2.123.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.123.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/HTTP_127.2.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:43:40 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563420, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:43:40.728508 2540 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:43:40.739122 2540 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:43:41.633107 2540 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:43:41.633173 2540 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms
I20260928 02:43:41.633186 2540 mini_ranger_kms.cc:227] Using postgres at 127.2.123.20:39231
2026-09-28 02:43:41,783 [I] DB FLAVOR :POSTGRES
2026-09-28 02:43:41,783 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:43:41,784 [I] Checking connection
2026-09-28 02:43:41,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:43:42,037 [I] connection success
2026-09-28 02:43:42,037 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:43:42,038 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:43:42,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:43:42,283 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:43:42,284 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:43:42,284 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:43:42,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.123.20:39231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:43:42,539 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:43:43.042992 2540 mini_ranger_kms.cc:310] Created kms service
I20260928 02:43:43.212299 2540 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:43:43.260119 2540 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:43:43 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563420, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:43.638725 2540 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:43:43.638993 2540 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCSnLDv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCSnLDv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:43:43.639019 2540 mini_ranger_kms.cc:240] Using host: 127.2.123.20
I20260928 02:43:43.642510 2540 mini_ranger_kms.cc:276] Ranger KMS PID: 14699
I20260928 02:43:43.642587 2540 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.123.20:46447
Sep 28, 2026 2:43:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:43:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:43:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:43:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:43:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:43:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:43:45.675576 2540 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.947s user 0.000s sys 0.136s
I20260928 02:43:45.675709 2540 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:43:45.675788 2540 mini_ranger_kms.cc:401] 127.2.123.20:46447/kms/v1/keys
Sep 28 02:43:45 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.123.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:43:46 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.123.20@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
Sep 28 02:43:46 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.123.20@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
Sep 28 02:43:48 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:43:48.907401 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45725
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:45725
--ranger_config_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:46447/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:49.017263 14750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:49.017535 14750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:49.017573 14750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:49.021652 14750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:49.021723 14750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:49.021740 14750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:49.021755 14750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:49.021770 14750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:49.027278 14750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.123.20:46447/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:45725
--ranger_config_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:49.028295 14750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:49.029320 14750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:49.038496 14756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:49.038477 14758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:49.038481 14755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:49.038599 14750 server_base.cc:1034] running on GCE node
I20260928 02:43:49.039336 14750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:49.040493 14750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:49.041659 14750 hybrid_clock.cc:648] HybridClock initialized: now 1790563429041639 us; error 33 us; skew 500 ppm
Sep 28 02:43:49 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:49.044466 14750 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:43:49.045339 14750 webserver.cc:460] Webserver started at http://127.2.123.62:34557/ using document root <none> and password file <none>
I20260928 02:43:49.045841 14750 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:49.045895 14750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:49.046101 14750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:43:49 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:49.105103 14750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "3891a0e8df7341679af253b5ae79ef1a"
format_stamp: "Formatted at 2026-09-28 02:43:49 on dist-test-slave-rqvb"
server_key: "5fe156f5c4af68686c35939cce0aff63"
server_key_iv: "3d65a23819b4bf46d28ef17006043320"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:49.105854 14750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "3891a0e8df7341679af253b5ae79ef1a"
format_stamp: "Formatted at 2026-09-28 02:43:49 on dist-test-slave-rqvb"
server_key: "5fe156f5c4af68686c35939cce0aff63"
server_key_iv: "3d65a23819b4bf46d28ef17006043320"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:49.110793 14750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 02:43:49.125424 14765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:49.126845 14750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:43:49.126977 14750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "3891a0e8df7341679af253b5ae79ef1a"
format_stamp: "Formatted at 2026-09-28 02:43:49 on dist-test-slave-rqvb"
server_key: "5fe156f5c4af68686c35939cce0aff63"
server_key_iv: "3d65a23819b4bf46d28ef17006043320"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:49.127105 14750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:49.140506 14750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:49.143515 14750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:49.143699 14750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:49.151093 14750 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:45725
I20260928 02:43:49.151109 14817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:45725 every 8 connection(s)
I20260928 02:43:49.152158 14750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:49.154788 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 14750
I20260928 02:43:49.154922 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:43:49.155227 2540 external_mini_cluster.cc:1428] Setting key 75cb7cdfee854242461fb9b6e420d549
I20260928 02:43:49.155238 14818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:49.162268 14818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Bootstrap starting.
I20260928 02:43:49.164811 14818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:49.165678 14818 log.cc:824] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:43:49 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:49.167940 14818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: No bootstrap required, opened a new log
I20260928 02:43:49.171949 14818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } }
I20260928 02:43:49.172240 14818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:49.172459 14818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3891a0e8df7341679af253b5ae79ef1a, State: Initialized, Role: FOLLOWER
I20260928 02:43:49.172991 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:49.156679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.2.123.62:45725)
0928 02:43:49.157169 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:49.157190 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:49.157237 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:43:49.158248 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:49.158273 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:49.158728 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:49.159128 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:49.159989 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:49.161517 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:49.162684 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:49.163415 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:49.163582 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:49.166746 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:49.166752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:49.166770 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:49.166810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:49.169471 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:49.170076 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:49.170082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:49.170089 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:49.170231 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:49.170541 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:49.170544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:49.170546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:49.171108 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:49.171311 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:49.171641 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:43:49.171937 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":152,"threads_started":1}
I20260928 02:43:49.172991 14818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } }
I20260928 02:43:49.173242 14818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:49.173331 14818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:49.173434 14818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:49.174593 14818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } }
I20260928 02:43:49.174945 14818 leader_election.cc:302] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3891a0e8df7341679af253b5ae79ef1a; no voters:
I20260928 02:43:49.175261 14818 leader_election.cc:288] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:49.175350 14823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:49.175916 14823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 1 LEADER]: Becoming Leader. State: Replica: 3891a0e8df7341679af253b5ae79ef1a, State: Running, Role: LEADER
I20260928 02:43:49.176288 14823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } }
I20260928 02:43:49.176734 14818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:49.177568 14823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3891a0e8df7341679af253b5ae79ef1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } } }
I20260928 02:43:49.178370 14823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:49.178257 14825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3891a0e8df7341679af253b5ae79ef1a. Latest consensus state: current_term: 1 leader_uuid: "3891a0e8df7341679af253b5ae79ef1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } } }
I20260928 02:43:49.178661 14825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:49.179046 14818 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:43:49.180164 14835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:49.185745 14835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:49.193185 14835 catalog_manager.cc:1347] Generated new cluster ID: 32c22298ae1346b398e1ba3a648c8b0f
I20260928 02:43:49.193245 14835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:49.203240 14835 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:49.204639 14835 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:49.210119 14835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Generated new TSK 0
I20260928 02:43:49.210948 14835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:43:49.889887 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:46447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:45725
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:50.032691 14863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:50.033048 14863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:50.033115 14863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:50.039001 14863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:50.039069 14863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:50.039147 14863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:43:50.044623 14863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.123.20:46447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:45725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:50.045706 14863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:50.046909 14863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:50.057128 14871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:50.057154 14868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:50.057250 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:50.057453 14863 server_base.cc:1034] running on GCE node
I20260928 02:43:50.060062 14863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:50.060889 14863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:50.062052 14863 hybrid_clock.cc:648] HybridClock initialized: now 1790563430062026 us; error 38 us; skew 500 ppm
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:50.065728 14863 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:43:50.066994 14863 webserver.cc:460] Webserver started at http://127.2.123.1:42371/ using document root <none> and password file <none>
I20260928 02:43:50.067718 14863 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:50.067804 14863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:50.068051 14863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:50.085193 14863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "3ef3213427b24f19a480330bb0fb4e9a"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "aa7c58aa2566f8e1e6aa31ae145591e7"
server_key_iv: "979ef31fb2fb737a8ce0ebe6462d0341"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.085748 14863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "3ef3213427b24f19a480330bb0fb4e9a"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "aa7c58aa2566f8e1e6aa31ae145591e7"
server_key_iv: "979ef31fb2fb737a8ce0ebe6462d0341"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.091183 14863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 02:43:50.113812 14880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.115417 14863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 02:43:50.115546 14863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "3ef3213427b24f19a480330bb0fb4e9a"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "aa7c58aa2566f8e1e6aa31ae145591e7"
server_key_iv: "979ef31fb2fb737a8ce0ebe6462d0341"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.115675 14863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:50.140956 14863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:50.150065 14863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:50.150256 14863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:50.151060 14863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:50.152123 14863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:50.152171 14863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.152221 14863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:50.152241 14863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.167587 14863 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:36267
I20260928 02:43:50.168727 14863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:50.170344 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 14863
I20260928 02:43:50.170460 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:50.170745 2540 external_mini_cluster.cc:1428] Setting key 805672800f4cd2cbcc801b843e7fbbcd
I20260928 02:43:50.178689 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:36267 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:50.195214 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.181182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45725 (local address 127.2.123.1:58971)
0928 02:43:50.181667 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:50.181719 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:50.183310 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:50.183918 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:50.183928 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:50.184268 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:50.184982 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:50.185004 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.186136 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.186140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:50.187208 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:50.187218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.187710 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.187715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:50.187898 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:50.188443 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:50.188460 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:50.189785 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:50.191826 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:50.191832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:50.191849 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:50.192182 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:50.192509 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:50.192512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:50.192514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:50.192694 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:50.193212 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:50.193218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:50.193546 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 02:43:50.193758 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 02:43:50.194147 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20260928 02:43:50.195977 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.180683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:58971 (local address 127.2.123.62:45725)
0928 02:43:50.181205 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:50.181213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:50.181840 (+ 627us) server_negotiation.cc:407] Connection header received
0928 02:43:50.183545 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:50.183548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:50.183598 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:50.183685 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:50.185155 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.185940 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.187337 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.187569 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.187658 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:50.189951 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:50.189954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:50.189978 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:50.190008 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:50.191680 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:50.192300 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:50.192304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:50.192305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:50.192401 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:50.192820 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:50.192823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:50.192825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:50.193051 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:50.193145 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:50.195670 (+ 2525us) server_negotiation.cc:299] Negotiation successful
0928 02:43:50.195813 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":93,"threads_started":1}
I20260928 02:43:50.197567 14994 heartbeater.cc:347] Connected to a master server at 127.2.123.62:45725
I20260928 02:43:50.197816 14994 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:50.198346 14994 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:50.199934 14782 ts_manager.cc:194] Registered new tserver with Master: 3ef3213427b24f19a480330bb0fb4e9a (127.2.123.1:36267)
I20260928 02:43:50.201071 14782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:58971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:43:50.224583 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:46447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:45725
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:43:50.352689 15002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:50.352952 15002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:50.353008 15002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:50.357702 15002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:50.357838 15002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:50.357945 15002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:43:50.363706 15002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.123.20:46447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:45725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:50.364954 15002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:50.365933 15002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:50.375046 15011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:50.375349 15008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:50.375058 15009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:50.376029 15002 server_base.cc:1034] running on GCE node
I20260928 02:43:50.376502 15002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:50.377156 15002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:50.378341 15002 hybrid_clock.cc:648] HybridClock initialized: now 1790563430378313 us; error 36 us; skew 500 ppm
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:50.381644 15002 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:43:50.382560 15002 webserver.cc:460] Webserver started at http://127.2.123.2:43391/ using document root <none> and password file <none>
I20260928 02:43:50.383087 15002 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:50.383149 15002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:50.383338 15002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:50.398442 15002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "818ba98f9799435696ed0d25ac69dcae"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "0d0a8f720901fb1225bae6c8396ce3d8"
server_key_iv: "6ee6317766821ba581ac7ece858f0709"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.399537 15002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "818ba98f9799435696ed0d25ac69dcae"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "0d0a8f720901fb1225bae6c8396ce3d8"
server_key_iv: "6ee6317766821ba581ac7ece858f0709"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.407476 15002 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260928 02:43:50.429575 15018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.438614 15002 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.000s
I20260928 02:43:50.438786 15002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "818ba98f9799435696ed0d25ac69dcae"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "0d0a8f720901fb1225bae6c8396ce3d8"
server_key_iv: "6ee6317766821ba581ac7ece858f0709"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.438959 15002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:50.455361 15002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:50.458868 15002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:50.459113 15002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:50.460045 15002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:50.461292 15002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:50.461404 15002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.461503 15002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:50.461565 15002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.471983 15002 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:40113
I20260928 02:43:50.473471 15002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:50.474769 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 15002
I20260928 02:43:50.474880 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:50.475168 2540 external_mini_cluster.cc:1428] Setting key 2720a558232bd1380f90cce21346c9f2
I20260928 02:43:50.487313 15131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:40113 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:43:50.521576 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.485644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:34889 (local address 127.2.123.62:45725)
0928 02:43:50.485774 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:50.485779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:50.492255 (+ 6476us) server_negotiation.cc:407] Connection header received
0928 02:43:50.493601 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:50.493605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:50.493664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:50.493717 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:50.495424 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.496228 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.497935 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.498327 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.498407 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:50.516810 (+ 18403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:50.516815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:50.516818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:50.516859 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:50.518757 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:50.519558 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:50.519562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:50.519564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:50.519657 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:50.520100 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:50.520103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:50.520105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:50.520344 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:50.520439 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:50.521255 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 02:43:50.521400 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:50.522677 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.486231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45725 (local address 127.2.123.2:34889)
0928 02:43:50.492067 (+ 5836us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:50.492132 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:50.493332 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:50.494192 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:50.494202 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:50.494561 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:50.495244 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:50.495274 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.496386 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.496389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:50.497777 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:50.497788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.498629 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.498633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:50.498833 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:50.508707 (+ 9874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:50.508735 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:50.516506 (+ 7771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:50.518909 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:50.518918 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:50.518962 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:50.519360 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:50.519760 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:50.519764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:50.519766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:50.519987 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:50.520495 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:50.520502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:50.520913 (+ 411us) client_negotiation.cc:769] Sending connection context
0928 02:43:50.521154 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 02:43:50.521532 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5573,"thread_start_us":155,"threads_started":1}
I20260928 02:43:50.524349 15132 heartbeater.cc:347] Connected to a master server at 127.2.123.62:45725
I20260928 02:43:50.524595 15132 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:50.525164 15132 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:50.526324 14782 ts_manager.cc:194] Registered new tserver with Master: 818ba98f9799435696ed0d25ac69dcae (127.2.123.2:40113)
I20260928 02:43:50.527082 14782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:34889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:43:50.544333 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:46447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:45725
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:50.693346 15137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:50.693797 15137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:50.693938 15137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:50.700134 15137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:50.700297 15137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:50.700439 15137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:43:50.708024 15137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.123.20:46447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:45725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:50.709690 15137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:50.711174 15137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:50.724848 15143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:50.725442 15142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:50.725741 15145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:50.726999 15137 server_base.cc:1034] running on GCE node
I20260928 02:43:50.727644 15137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:50.728508 15137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:50.729740 15137 hybrid_clock.cc:648] HybridClock initialized: now 1790563430729667 us; error 95 us; skew 500 ppm
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:50.733721 15137 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:43:50.734958 15137 webserver.cc:460] Webserver started at http://127.2.123.3:41623/ using document root <none> and password file <none>
I20260928 02:43:50.735736 15137 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:50.735886 15137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:50.736200 15137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:50.759464 15137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "e17ee7e5f96a490485e131852402baf1"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "a1f3707e758150f918e65fc2fed8484d"
server_key_iv: "a65fb4bd4be4a754e3f77d8414e08d43"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.761289 15137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "e17ee7e5f96a490485e131852402baf1"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "a1f3707e758150f918e65fc2fed8484d"
server_key_iv: "a65fb4bd4be4a754e3f77d8414e08d43"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.767117 15137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:43:50.791395 15152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.793217 15137 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 02:43:50.793365 15137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "e17ee7e5f96a490485e131852402baf1"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "a1f3707e758150f918e65fc2fed8484d"
server_key_iv: "a65fb4bd4be4a754e3f77d8414e08d43"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:50.793497 15137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:50.824990 15137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:50.829360 15137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:50.829635 15137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:50.830516 15137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:50.831755 15137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:50.831858 15137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.831954 15137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:50.832019 15137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:50.841979 15137 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:43431
I20260928 02:43:50.843196 15137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:50.843940 15265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:43431 every 8 connection(s)
I20260928 02:43:50.849112 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 15137
I20260928 02:43:50.849259 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:50.849612 2540 external_mini_cluster.cc:1428] Setting key 8bd95a545fab7ad332cc75e8d4f26267
Sep 28 02:43:50 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:50.864290 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.848770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:43141 (local address 127.2.123.62:45725)
0928 02:43:50.848925 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:50.848931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:50.849433 (+ 502us) server_negotiation.cc:407] Connection header received
0928 02:43:50.851194 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:50.851198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:50.851255 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:50.851312 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:50.853047 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.854010 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.856010 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.856379 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.856466 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:50.859731 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:50.859736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:50.859739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:50.859770 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:50.861516 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:50.862229 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:50.862233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:50.862236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:50.862337 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:50.862784 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:50.862788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:50.862790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:50.863048 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:50.863143 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:50.863958 (+ 815us) server_negotiation.cc:299] Negotiation successful
0928 02:43:50.864093 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:43:50.866261 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.847572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45725 (local address 127.2.123.3:43141)
0928 02:43:50.849251 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:50.849316 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:50.850927 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:50.851756 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:50.851765 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:50.852144 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:50.852842 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:50.852864 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.854412 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.854416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:50.855853 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:50.855866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:50.856687 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:50.856691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:50.856900 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:50.857595 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:50.857619 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:50.859507 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:50.861662 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:50.861669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:50.861687 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:50.862084 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:50.862458 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:50.862461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:50.862463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:50.862657 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:50.863259 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:50.863266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:50.863638 (+ 372us) client_negotiation.cc:769] Sending connection context
0928 02:43:50.864758 (+ 1120us) client_negotiation.cc:240] Negotiation successful
0928 02:43:50.865143 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"thread_start_us":156,"threads_started":1}
I20260928 02:43:50.867779 15266 heartbeater.cc:347] Connected to a master server at 127.2.123.62:45725
I20260928 02:43:50.868040 15266 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:50.868669 15266 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:50.870004 14782 ts_manager.cc:194] Registered new tserver with Master: e17ee7e5f96a490485e131852402baf1 (127.2.123.3:43431)
I20260928 02:43:50.870743 14782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:43141
I20260928 02:43:50.875506 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:43:50.876356 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 14863
I20260928 02:43:50.882692 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 15002
I20260928 02:43:50.889288 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 15137
I20260928 02:43:50.896046 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 14750
I20260928 02:43:50.903050 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45725
--webserver_interface=127.2.123.62
--webserver_port=34557
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:45725
--ranger_config_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:46447/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.123.20:46447/kms with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:51.014345 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:51.014600 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:51.014642 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:51.018692 15269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:51.018750 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:51.018769 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:51.018784 15269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:51.018801 15269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:51.024124 15269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.123.20:46447/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:45725
--ranger_config_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:51.025106 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:51.026125 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:51.034235 15275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:51.034250 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:51.034423 15269 server_base.cc:1034] running on GCE node
W20260928 02:43:51.034269 15277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:51.035091 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:51.036187 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:51.037359 15269 hybrid_clock.cc:648] HybridClock initialized: now 1790563431037342 us; error 29 us; skew 500 ppm
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:51.040143 15269 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:43:51.040992 15269 webserver.cc:460] Webserver started at http://127.2.123.62:34557/ using document root <none> and password file <none>
I20260928 02:43:51.041488 15269 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:51.041555 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:51.112600 15269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:43:51.114934 15285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.116194 15269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:43:51.116315 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "3891a0e8df7341679af253b5ae79ef1a"
format_stamp: "Formatted at 2026-09-28 02:43:49 on dist-test-slave-rqvb"
server_key: "5fe156f5c4af68686c35939cce0aff63"
server_key_iv: "3d65a23819b4bf46d28ef17006043320"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:51.116901 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:51.138149 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:51.140974 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:51.141147 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:51.148053 15269 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:45725
I20260928 02:43:51.148074 15337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:45725 every 8 connection(s)
I20260928 02:43:51.149113 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:51.152726 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 15269
I20260928 02:43:51.153198 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:46447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:36267
--local_ip_for_outbound_sockets=127.2.123.1
--tserver_master_addrs=127.2.123.62:45725
--webserver_port=42371
--webserver_interface=127.2.123.1
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.123.20:46447/kms with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:51.153499 15338 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:43:51.155421 15338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Bootstrap starting.
I20260928 02:43:51.171411 15338 log.cc:824] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:51.177907 15338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:43:51.178308 15338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Bootstrap complete.
I20260928 02:43:51.182328 15338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } }
I20260928 02:43:51.182760 15338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3891a0e8df7341679af253b5ae79ef1a, State: Initialized, Role: FOLLOWER
I20260928 02:43:51.183261 15338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } }
I20260928 02:43:51.183374 15338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:51.183432 15338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:51.183521 15338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:43:51.185385 15338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } }
I20260928 02:43:51.185693 15338 leader_election.cc:302] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3891a0e8df7341679af253b5ae79ef1a; no voters:
I20260928 02:43:51.186015 15338 leader_election.cc:288] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:43:51.186132 15342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:43:51.186487 15342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [term 2 LEADER]: Becoming Leader. State: Replica: 3891a0e8df7341679af253b5ae79ef1a, State: Running, Role: LEADER
I20260928 02:43:51.186947 15342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } }
I20260928 02:43:51.187148 15338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:51.188446 15344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3891a0e8df7341679af253b5ae79ef1a. Latest consensus state: current_term: 2 leader_uuid: "3891a0e8df7341679af253b5ae79ef1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } } }
I20260928 02:43:51.188491 15343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3891a0e8df7341679af253b5ae79ef1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3891a0e8df7341679af253b5ae79ef1a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45725 } } }
I20260928 02:43:51.188570 15344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:51.188593 15343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:51.189347 15338 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:43:51.190425 15353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:51.196013 15353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:51.196667 15353 catalog_manager.cc:1259] Loaded cluster ID: 32c22298ae1346b398e1ba3a648c8b0f
I20260928 02:43:51.196875 15353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:51.199445 15353 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:51.200171 15353 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Loaded TSK: 0
I20260928 02:43:51.201491 15353 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:43:51.295892 15340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:51.296662 15340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:51.296835 15340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:51.303117 15340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:51.303194 15340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:51.303282 15340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:43:51.311156 15340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.123.20:46447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:36267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=42371
--tserver_master_addrs=127.2.123.62:45725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:51.312561 15340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:51.313920 15340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:51.326907 15376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:51.326763 15375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:51.327291 15340 server_base.cc:1034] running on GCE node
W20260928 02:43:51.327842 15378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:51.328383 15340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:51.329196 15340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:51.330374 15340 hybrid_clock.cc:648] HybridClock initialized: now 1790563431330335 us; error 55 us; skew 500 ppm
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:51.337199 15340 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:43:51.338383 15340 webserver.cc:460] Webserver started at http://127.2.123.1:42371/ using document root <none> and password file <none>
I20260928 02:43:51.339136 15340 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:51.339242 15340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:51.357916 15340 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:43:51.361099 15386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.362587 15340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:51.362706 15340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "3ef3213427b24f19a480330bb0fb4e9a"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "aa7c58aa2566f8e1e6aa31ae145591e7"
server_key_iv: "979ef31fb2fb737a8ce0ebe6462d0341"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:51.363253 15340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:51.378928 15340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:51.382704 15340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:51.382944 15340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:51.383690 15340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:51.384811 15340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:51.384907 15340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.384979 15340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:51.385021 15340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.395963 15340 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:36267
I20260928 02:43:51.395998 15499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:36267 every 8 connection(s)
I20260928 02:43:51.397305 15340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:51.400554 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 15340
I20260928 02:43:51.401041 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:46447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:40113
--local_ip_for_outbound_sockets=127.2.123.2
--tserver_master_addrs=127.2.123.62:45725
--webserver_port=43391
--webserver_interface=127.2.123.2
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.123.20:46447/kms with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:51.417363 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.399521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45725 (local address 127.2.123.1:52993)
0928 02:43:51.400643 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:51.400697 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:51.401757 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:51.403324 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:51.403334 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:51.403669 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:51.404322 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:51.404342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.406124 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.406127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:51.407529 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:51.407540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.408783 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.408787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:51.408971 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:51.409638 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:51.409660 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:51.411381 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:51.413834 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:51.413841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:51.413856 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:51.414210 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:51.414600 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:51.414603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:51.414605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:51.414785 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:51.415533 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:51.415540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:51.415850 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 02:43:51.416045 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 02:43:51.416370 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":128,"threads_started":1}
I20260928 02:43:51.419083 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.399510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:52993 (local address 127.2.123.62:45725)
0928 02:43:51.400005 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:51.400030 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:51.402103 (+ 2073us) server_negotiation.cc:407] Connection header received
0928 02:43:51.402421 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:51.402438 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:51.402868 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:51.403227 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:51.404514 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.405992 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.407668 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.408395 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.408551 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:51.411610 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:51.411617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:51.411634 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:51.411666 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:51.413681 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:51.414348 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:51.414355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:51.414360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:51.414495 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:51.414892 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:51.414895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:51.414897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:51.415358 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:51.415482 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:51.417865 (+ 2383us) server_negotiation.cc:299] Negotiation successful
0928 02:43:51.418208 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
I20260928 02:43:51.421936 15500 heartbeater.cc:347] Connected to a master server at 127.2.123.62:45725
I20260928 02:43:51.422272 15500 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.422905 15500 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
W20260928 02:43:51.425127 15500 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:45725 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:43:51.425344 15500 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.425575 15500 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:51.426425 15500 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.426585 15500 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
W20260928 02:43:51.427284 15500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:43:51.542085 15504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:51.542338 15504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:51.542380 15504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:51.546475 15504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:51.546530 15504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:51.546591 15504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:43:51.553412 15504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.123.20:46447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:40113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=43391
--tserver_master_addrs=127.2.123.62:45725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:51.554683 15504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:51.556015 15504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:51.566812 15509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:51.566820 15510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:51.567265 15512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:51.567600 15504 server_base.cc:1034] running on GCE node
I20260928 02:43:51.567998 15504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:51.568604 15504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:51.569780 15504 hybrid_clock.cc:648] HybridClock initialized: now 1790563431569754 us; error 55 us; skew 500 ppm
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:51.573259 15504 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:43:51.574339 15504 webserver.cc:460] Webserver started at http://127.2.123.2:43391/ using document root <none> and password file <none>
I20260928 02:43:51.574901 15504 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:51.574971 15504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:51.606674 15504 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.003s sys 0.000s
I20260928 02:43:51.610028 15520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.611776 15504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 02:43:51.611904 15504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "818ba98f9799435696ed0d25ac69dcae"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "0d0a8f720901fb1225bae6c8396ce3d8"
server_key_iv: "6ee6317766821ba581ac7ece858f0709"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:51.612622 15504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:51.630836 15504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:51.634143 15504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:51.634369 15504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:51.634948 15504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:51.635859 15504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:51.635915 15504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.635993 15504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:51.636023 15504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.646191 15504 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:40113
I20260928 02:43:51.646235 15633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:40113 every 8 connection(s)
I20260928 02:43:51.647313 15504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:51.650738 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 15504
I20260928 02:43:51.651176 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.123.20:46447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:43431
--local_ip_for_outbound_sockets=127.2.123.3
--tserver_master_addrs=127.2.123.62:45725
--webserver_port=41623
--webserver_interface=127.2.123.3
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.123.20:46447/kms with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:51.662256 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.649589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45725 (local address 127.2.123.2:37297)
0928 02:43:51.650081 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:51.650131 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:51.651168 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:51.651947 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:51.651956 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:51.652288 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:51.653066 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:51.653085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.654172 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.654175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:51.655051 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:51.655059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.655533 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.655537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:51.655699 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:51.656234 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:51.656250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:51.657658 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:51.659352 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:51.659357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:51.659369 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:51.659638 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:51.659903 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:51.659906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:51.659908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:51.660043 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:51.660412 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:51.660418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:51.660662 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 02:43:51.660832 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:43:51.661146 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260928 02:43:51.664270 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.649153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:37297 (local address 127.2.123.62:45725)
0928 02:43:51.649306 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:51.649310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:51.650245 (+ 935us) server_negotiation.cc:407] Connection header received
0928 02:43:51.651428 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:51.651431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:51.651488 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:51.651550 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:51.653245 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.654038 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.655170 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.655396 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.655483 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:51.657815 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:51.657818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:51.657820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:51.657848 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:51.659231 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:51.659740 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:51.659744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:51.659745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:51.659825 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:51.660127 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:51.660130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:51.660132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:51.660318 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:51.660400 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:51.660898 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 02:43:51.661028 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:43:51.665148 15634 heartbeater.cc:347] Connected to a master server at 127.2.123.62:45725
I20260928 02:43:51.665453 15634 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.665951 15634 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
W20260928 02:43:51.667443 15634 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:45725 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:43:51.667553 15634 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.667711 15634 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:51.668429 15634 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.668581 15634 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
W20260928 02:43:51.669256 15634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:43:51.793164 15637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:51.793514 15637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:51.793584 15637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:51.797719 15637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:51.797770 15637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:51.797829 15637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:43:51.803449 15637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.123.20:46447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:43431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=41623
--tserver_master_addrs=127.2.123.62:45725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:51.804451 15637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:51.805441 15637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:51.816214 15644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:51.816262 15646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:51.816215 15643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:51.816872 15637 server_base.cc:1034] running on GCE node
I20260928 02:43:51.817342 15637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:51.818104 15637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:51.819279 15637 hybrid_clock.cc:648] HybridClock initialized: now 1790563431819211 us; error 82 us; skew 500 ppm
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:51.822669 15637 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:43:51.823736 15637 webserver.cc:460] Webserver started at http://127.2.123.3:41623/ using document root <none> and password file <none>
I20260928 02:43:51.824304 15637 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:51.824366 15637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:51.844375 15637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:51.847357 15654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.848888 15637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:51.849012 15637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "e17ee7e5f96a490485e131852402baf1"
format_stamp: "Formatted at 2026-09-28 02:43:50 on dist-test-slave-rqvb"
server_key: "a1f3707e758150f918e65fc2fed8484d"
server_key_iv: "a65fb4bd4be4a754e3f77d8414e08d43"
server_key_version: "kuduclusterkey@0"
I20260928 02:43:51.849685 15637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:51.872334 15637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:51.875698 15637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:51.875887 15637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:51.876642 15637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:51.877723 15637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:51.877839 15637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.877941 15637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:51.878041 15637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:51.890579 15637 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:43431
I20260928 02:43:51.890607 15767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:43431 every 8 connection(s)
I20260928 02:43:51.892373 15637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:51.894405 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 15637
Sep 28 02:43:51 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:51.908478 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.896067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45725 (local address 127.2.123.3:55265)
0928 02:43:51.897132 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:51.897180 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:51.898114 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:51.898777 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:51.898784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:51.899018 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:51.899486 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:51.899499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.900240 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.900243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:51.901347 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:51.901355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.901953 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.901978 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:51.902136 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:51.902610 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:51.902626 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:51.903844 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:51.905395 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:51.905400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:51.905415 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:51.905662 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:51.905901 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:51.905904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:51.905907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:51.906124 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:51.906607 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:51.906614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:51.906924 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 02:43:51.907118 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 02:43:51.907449 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":135,"threads_started":1}
I20260928 02:43:51.909170 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.895593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:55265 (local address 127.2.123.62:45725)
0928 02:43:51.895877 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:51.895881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:51.897266 (+ 1385us) server_negotiation.cc:407] Connection header received
0928 02:43:51.898408 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:51.898411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:51.898459 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:51.898547 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:51.899618 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.900129 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.901460 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.901795 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.901916 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:51.903992 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:51.903996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:51.903999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:51.904023 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:51.905295 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:51.905748 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:51.905751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:51.905753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:51.905826 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:51.906243 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:51.906246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:51.906248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:51.906463 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:51.906556 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:51.908871 (+ 2315us) server_negotiation.cc:299] Negotiation successful
0928 02:43:51.908996 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260928 02:43:51.910955 15768 heartbeater.cc:347] Connected to a master server at 127.2.123.62:45725
I20260928 02:43:51.911226 15768 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.911795 15768 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
W20260928 02:43:51.914048 15768 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:45725 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:43:51.914178 15768 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.914358 15768 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:51.915197 15768 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:51.915364 15768 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
W20260928 02:43:51.916087 15768 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:43:51.916337 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.895131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.2.123.62:45725)
0928 02:43:51.895260 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:51.895264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:51.896179 (+ 915us) server_negotiation.cc:407] Connection header received
0928 02:43:51.896364 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:51.896367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:51.896418 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:51.896492 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:51.897539 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.898358 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.900628 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:51.900979 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:51.901072 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:51.912198 (+ 11126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:51.912202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:51.912205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:51.912229 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:51.913709 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:51.914226 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:51.914230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:51.914231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:51.914308 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:51.915323 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:51.915326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:51.915327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:51.915531 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:51.915721 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:51.916087 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 02:43:51.916207 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:43:52 dist-test-slave-rqvb krb5kdc[14306](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:52.427495 15500 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:52.427788 15500 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:52.428891 15302 ts_manager.cc:194] Registered new tserver with Master: 3ef3213427b24f19a480330bb0fb4e9a (127.2.123.1:36267)
I20260928 02:43:52.430608 15302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:52993
Sep 28 02:43:52 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:52.669469 15634 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:52.669740 15634 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:52.670547 15302 ts_manager.cc:194] Registered new tserver with Master: 818ba98f9799435696ed0d25ac69dcae (127.2.123.2:40113)
I20260928 02:43:52.671330 15302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:37297
I20260928 02:43:52.916339 15768 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:52.916661 15768 heartbeater.cc:511] Master 127.2.123.62:45725 requested a full tablet report, sending...
I20260928 02:43:52.917537 15302 ts_manager.cc:194] Registered new tserver with Master: e17ee7e5f96a490485e131852402baf1 (127.2.123.3:43431)
I20260928 02:43:52.918314 15302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:55265
I20260928 02:43:52.923599 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:43:52.938862 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.926272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.2.123.62:45725)
0928 02:43:52.930115 (+ 3843us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:52.930119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:52.930134 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:43:52.930176 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:52.930179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:52.930229 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:52.930319 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:52.930991 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:52.931524 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:52.932715 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:52.933062 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:52.933164 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:52.935006 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:52.935010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:52.935013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:52.935039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:52.936718 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:52.937164 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:52.937168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:52.937170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:52.937253 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:52.937509 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:52.937512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:52.937514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:52.937720 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:52.937828 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:52.938559 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 02:43:52.938693 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3765,"thread_start_us":89,"threads_started":1}
I20260928 02:43:52.945196 15302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:52.951757 15302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 28 02:43:52 dist-test-slave-rqvb krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for HTTP/127.2.123.20@KRBTEST.COM
I20260928 02:43:52.993650 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.979933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:36267 (local address 127.0.0.1:56066)
0928 02:43:52.984865 (+ 4932us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:52.984933 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:52.985140 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:52.986641 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:52.986648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:52.986673 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:52.986813 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:52.986820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:52.989198 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:52.989202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:52.990924 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:52.990936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:52.993068 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:52.993072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:52.993193 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:52.993232 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:43:52.993336 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:43:52.993472 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3584,"thread_start_us":125,"threads_started":1}
I20260928 02:43:52.994554 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.980991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.2.123.1:36267)
0928 02:43:52.983804 (+ 2813us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:52.983812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:52.985299 (+ 1487us) server_negotiation.cc:407] Connection header received
0928 02:43:52.985381 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:52.985386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:52.985639 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:52.985782 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:52.986966 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:52.989073 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:52.991069 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:52.992752 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:52.992858 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:52.993010 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:52.994304 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0928 02:43:52.994413 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2699,"thread_start_us":98,"threads_started":1}
I20260928 02:43:52.998046 15425 tablet_service.cc:1467] Processing CreateTablet for tablet ca273b5e96c64d5795515784b207b1fd (DEFAULT_TABLE table=test-table [id=9be23d682db748878c781e83ac7b7dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:52.999344 15425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca273b5e96c64d5795515784b207b1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:53.011794 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.983169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:43431 (local address 127.0.0.1:48520)
0928 02:43:52.986196 (+ 3027us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:52.986207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:52.986280 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:53.007486 (+ 21206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:53.007489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:53.007511 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:53.007608 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:53.007624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.009074 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.009079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:53.010312 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:53.010324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.011371 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.011375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:53.011466 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.011495 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:43:53.011547 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:43:53.011629 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2970,"mutex_wait_us":30,"thread_start_us":82,"threads_started":1}
I20260928 02:43:53.012351 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.982874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:40113 (local address 127.0.0.1:60954)
0928 02:43:52.985923 (+ 3049us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:52.985937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:52.986029 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:53.006743 (+ 20714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:53.006747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:53.006774 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:53.006888 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:53.006895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.009149 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.009153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:53.010384 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:53.010395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.011520 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.011524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:53.011633 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.011658 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:43:53.011703 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:43:53.011772 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260928 02:43:53.012372 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.983046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.2.123.2:40113)
0928 02:43:53.006210 (+ 23164us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:53.006217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:53.006245 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:43:53.006329 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:53.006334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:53.006549 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:53.006732 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:53.007039 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.008127 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.010542 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.011376 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.011875 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.012027 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:53.012078 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:43:53.012171 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23035,"thread_start_us":117,"threads_started":1}
I20260928 02:43:53.013192 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.983515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.2.123.3:43431)
0928 02:43:53.007044 (+ 23529us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:53.007050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:53.007072 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:43:53.007145 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:53.007150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:53.007331 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:53.007764 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:53.007901 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.008938 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.010455 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.011234 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.012794 (+ 1560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.012934 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:53.012980 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:43:53.013066 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23419,"thread_start_us":108,"threads_started":1}
I20260928 02:43:53.014340 15803 tablet_bootstrap.cc:492] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a: Bootstrap starting.
I20260928 02:43:53.015558 15556 tablet_service.cc:1467] Processing CreateTablet for tablet ca273b5e96c64d5795515784b207b1fd (DEFAULT_TABLE table=test-table [id=9be23d682db748878c781e83ac7b7dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:53.015810 15702 tablet_service.cc:1467] Processing CreateTablet for tablet ca273b5e96c64d5795515784b207b1fd (DEFAULT_TABLE table=test-table [id=9be23d682db748878c781e83ac7b7dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:53.016734 15702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca273b5e96c64d5795515784b207b1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:53.016734 15556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca273b5e96c64d5795515784b207b1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:53.026422 15804 tablet_bootstrap.cc:492] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Bootstrap starting.
I20260928 02:43:53.027238 15803 tablet_bootstrap.cc:654] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:53.028184 15803 log.cc:824] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:53.029847 15804 tablet_bootstrap.cc:654] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:53.030678 15804 log.cc:824] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:53.031394 15805 tablet_bootstrap.cc:492] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Bootstrap starting.
I20260928 02:43:53.031833 15803 tablet_bootstrap.cc:492] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a: No bootstrap required, opened a new log
I20260928 02:43:53.032081 15803 ts_tablet_manager.cc:1397] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260928 02:43:53.033121 15804 tablet_bootstrap.cc:492] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: No bootstrap required, opened a new log
I20260928 02:43:53.033342 15804 ts_tablet_manager.cc:1397] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 02:43:53.033605 15805 tablet_bootstrap.cc:654] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:53.034436 15805 log.cc:824] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:53.036033 15803 raft_consensus.cc:348] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.037124 15804 raft_consensus.cc:348] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.037436 15804 raft_consensus.cc:374] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:53.037520 15804 raft_consensus.cc:729] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818ba98f9799435696ed0d25ac69dcae, State: Initialized, Role: FOLLOWER
I20260928 02:43:53.038113 15804 consensus_queue.cc:260] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.039290 15804 ts_tablet_manager.cc:1428] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:43:53.039698 15634 heartbeater.cc:503] Master 127.2.123.62:45725 was elected leader, sending a full tablet report...
I20260928 02:43:53.042027 15803 raft_consensus.cc:374] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:53.042178 15803 raft_consensus.cc:729] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef3213427b24f19a480330bb0fb4e9a, State: Initialized, Role: FOLLOWER
I20260928 02:43:53.042733 15803 consensus_queue.cc:260] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.043852 15803 ts_tablet_manager.cc:1428] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260928 02:43:53.044238 15500 heartbeater.cc:503] Master 127.2.123.62:45725 was elected leader, sending a full tablet report...
I20260928 02:43:53.065333 15805 tablet_bootstrap.cc:492] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: No bootstrap required, opened a new log
I20260928 02:43:53.078294 15805 ts_tablet_manager.cc:1397] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.000s
I20260928 02:43:53.083463 15805 raft_consensus.cc:348] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.083945 15805 raft_consensus.cc:374] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:53.084151 15805 raft_consensus.cc:729] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17ee7e5f96a490485e131852402baf1, State: Initialized, Role: FOLLOWER
I20260928 02:43:53.084825 15805 consensus_queue.cc:260] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.086163 15805 ts_tablet_manager.cc:1428] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:43:53.086618 15768 heartbeater.cc:503] Master 127.2.123.62:45725 was elected leader, sending a full tablet report...
W20260928 02:43:53.145162 15769 tablet.cc:2369] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:43:53.149489 15635 tablet.cc:2369] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:43:53.149487 15501 tablet.cc:2369] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:43:53.238233 15810 raft_consensus.cc:482] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:53.238461 15810 raft_consensus.cc:504] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.239677 15810 leader_election.cc:288] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818ba98f9799435696ed0d25ac69dcae (127.2.123.2:40113), e17ee7e5f96a490485e131852402baf1 (127.2.123.3:43431)
I20260928 02:43:53.244845 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.240532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:57495 (local address 127.2.123.3:43431)
0928 02:43:53.240672 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:53.240676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:53.240819 (+ 143us) server_negotiation.cc:407] Connection header received
0928 02:43:53.240921 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:53.240923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:53.240965 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:53.241041 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:53.241297 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.241994 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.243482 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.244109 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.244216 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.244284 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:53.244630 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 02:43:53.244710 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:43:53.244839 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.240050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:54243 (local address 127.2.123.2:40113)
0928 02:43:53.240183 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:53.240187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:53.240372 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:43:53.240530 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:53.240534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:53.240590 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:53.240681 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:53.241025 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.241771 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.243482 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.244109 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.244216 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.244324 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:53.244630 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 02:43:53.244710 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:43:53.244902 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.240403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:43431 (local address 127.2.123.1:57495)
0928 02:43:53.240717 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:53.240730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:53.240837 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:53.241067 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:53.241070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:53.241085 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:53.241184 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:53.241195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.242163 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.242166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:53.243365 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:53.243372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.244252 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.244257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:53.244423 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.244507 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 02:43:53.244587 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:43:53.244694 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":126,"thread_start_us":120,"threads_started":1}
I20260928 02:43:53.244907 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.239950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:40113 (local address 127.2.123.1:54243)
0928 02:43:53.240276 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:53.240292 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:53.240414 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:53.240711 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:53.240714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:53.240736 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:53.240915 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:53.240923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.241895 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.241899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:53.243334 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:53.243347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.244252 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.244257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:53.244445 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.244501 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:43:53.244607 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:43:53.244683 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":92,"thread_start_us":126,"threads_started":1}
I20260928 02:43:53.245661 15715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca273b5e96c64d5795515784b207b1fd" candidate_uuid: "3ef3213427b24f19a480330bb0fb4e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17ee7e5f96a490485e131852402baf1" is_pre_election: true
I20260928 02:43:53.245661 15587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca273b5e96c64d5795515784b207b1fd" candidate_uuid: "3ef3213427b24f19a480330bb0fb4e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818ba98f9799435696ed0d25ac69dcae" is_pre_election: true
I20260928 02:43:53.246018 15715 raft_consensus.cc:2454] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef3213427b24f19a480330bb0fb4e9a in term 0.
I20260928 02:43:53.246018 15587 raft_consensus.cc:2454] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef3213427b24f19a480330bb0fb4e9a in term 0.
I20260928 02:43:53.246546 15388 leader_election.cc:302] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef3213427b24f19a480330bb0fb4e9a, e17ee7e5f96a490485e131852402baf1; no voters:
I20260928 02:43:53.246805 15810 raft_consensus.cc:2781] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:53.246888 15810 raft_consensus.cc:482] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:53.246941 15810 raft_consensus.cc:3037] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:53.248461 15810 raft_consensus.cc:504] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.248809 15810 leader_election.cc:288] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [CANDIDATE]: Term 1 election: Requested vote from peers 818ba98f9799435696ed0d25ac69dcae (127.2.123.2:40113), e17ee7e5f96a490485e131852402baf1 (127.2.123.3:43431)
I20260928 02:43:53.249174 15587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca273b5e96c64d5795515784b207b1fd" candidate_uuid: "3ef3213427b24f19a480330bb0fb4e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818ba98f9799435696ed0d25ac69dcae"
I20260928 02:43:53.249241 15715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca273b5e96c64d5795515784b207b1fd" candidate_uuid: "3ef3213427b24f19a480330bb0fb4e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17ee7e5f96a490485e131852402baf1"
I20260928 02:43:53.249322 15587 raft_consensus.cc:3037] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:53.249354 15715 raft_consensus.cc:3037] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:53.250655 15715 raft_consensus.cc:2454] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef3213427b24f19a480330bb0fb4e9a in term 1.
I20260928 02:43:53.250655 15587 raft_consensus.cc:2454] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef3213427b24f19a480330bb0fb4e9a in term 1.
I20260928 02:43:53.250967 15388 leader_election.cc:302] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef3213427b24f19a480330bb0fb4e9a, e17ee7e5f96a490485e131852402baf1; no voters:
I20260928 02:43:53.251133 15810 raft_consensus.cc:2781] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:53.251353 15810 raft_consensus.cc:686] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [term 1 LEADER]: Becoming Leader. State: Replica: 3ef3213427b24f19a480330bb0fb4e9a, State: Running, Role: LEADER
I20260928 02:43:53.251673 15810 consensus_queue.cc:237] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } }
I20260928 02:43:53.254845 15302 catalog_manager.cc:5495] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ef3213427b24f19a480330bb0fb4e9a (127.2.123.1). New cstate: current_term: 1 leader_uuid: "3ef3213427b24f19a480330bb0fb4e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef3213427b24f19a480330bb0fb4e9a" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 36267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 } health_report { overall_health: UNKNOWN } } }
I20260928 02:43:53.280246 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.276866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.2.123.1:36267)
0928 02:43:53.276987 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:53.276992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:53.277005 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:43:53.277130 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:53.277132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:53.277188 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:53.277320 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:53.277538 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.278047 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.278863 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:53.279075 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:53.279158 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:53.279443 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:53.279858 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:53.279968 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:53.280033 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:43:53.280090 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:43:53.286042 15587 raft_consensus.cc:1261] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 1 FOLLOWER]: Refusing update from remote peer 3ef3213427b24f19a480330bb0fb4e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:53.286042 15715 raft_consensus.cc:1261] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 1 FOLLOWER]: Refusing update from remote peer 3ef3213427b24f19a480330bb0fb4e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:53.286736 15810 consensus_queue.cc:1035] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e17ee7e5f96a490485e131852402baf1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 43431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:53.286911 15821 consensus_queue.cc:1035] T ca273b5e96c64d5795515784b207b1fd P 3ef3213427b24f19a480330bb0fb4e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818ba98f9799435696ed0d25ac69dcae" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:53.293116 15825 mvcc.cc:205] Tried to move back new op lower bound from 7334147822732656640 to 7334147822601547776. Current Snapshot: MvccSnapshot[applied={T|T < 7334147822732656640}]
I20260928 02:43:53.294869 15826 mvcc.cc:205] Tried to move back new op lower bound from 7334147822732656640 to 7334147822601547776. Current Snapshot: MvccSnapshot[applied={T|T < 7334147822732656640}]
I20260928 02:43:53.302585 15302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:53.302752 15302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:53.305061 15302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3891a0e8df7341679af253b5ae79ef1a: Sending DeleteTablet for 3 replicas of tablet ca273b5e96c64d5795515784b207b1fd
I20260928 02:43:53.305935 15702 tablet_service.cc:1514] Processing DeleteTablet for tablet ca273b5e96c64d5795515784b207b1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:53 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:48520
I20260928 02:43:53.306093 15556 tablet_service.cc:1514] Processing DeleteTablet for tablet ca273b5e96c64d5795515784b207b1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:53 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:60954
I20260928 02:43:53.306378 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 15340
I20260928 02:43:53.307153 15835 tablet_replica.cc:330] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: stopping tablet replica
I20260928 02:43:53.307466 15835 raft_consensus.cc:2229] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:53.307740 15834 tablet_replica.cc:330] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: stopping tablet replica
I20260928 02:43:53.307840 15835 raft_consensus.cc:2258] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:53.308079 15834 raft_consensus.cc:2229] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:53.308378 15834 raft_consensus.cc:2258] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:53.309558 15835 ts_tablet_manager.cc:1905] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:53.309633 15834 ts_tablet_manager.cc:1905] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:53.311764 15834 ts_tablet_manager.cc:1918] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:53.311851 15834 log.cc:1196] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/ca273b5e96c64d5795515784b207b1fd
I20260928 02:43:53.312150 15834 ts_tablet_manager.cc:1939] T ca273b5e96c64d5795515784b207b1fd P e17ee7e5f96a490485e131852402baf1: Deleting consensus metadata
I20260928 02:43:53.312250 15835 ts_tablet_manager.cc:1918] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:53.312350 15835 log.cc:1196] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/ca273b5e96c64d5795515784b207b1fd
I20260928 02:43:53.312654 15835 ts_tablet_manager.cc:1939] T ca273b5e96c64d5795515784b207b1fd P 818ba98f9799435696ed0d25ac69dcae: Deleting consensus metadata
W20260928 02:43:53.313573 15287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:36267 (error 108)
I20260928 02:43:53.313736 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 15504
I20260928 02:43:53.314095 15287 catalog_manager.cc:4841] TS e17ee7e5f96a490485e131852402baf1 (127.2.123.3:43431): tablet ca273b5e96c64d5795515784b207b1fd (table test-table [id=9be23d682db748878c781e83ac7b7dcf]) successfully deleted
I20260928 02:43:53.314262 15287 catalog_manager.cc:4841] TS 818ba98f9799435696ed0d25ac69dcae (127.2.123.2:40113): tablet ca273b5e96c64d5795515784b207b1fd (table test-table [id=9be23d682db748878c781e83ac7b7dcf]) successfully deleted
I20260928 02:43:53.315663 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.315366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:36267 (local address 127.0.0.1:56090)
0928 02:43:53.315501 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:53.315577 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:43:53.315824 15287 catalog_manager.cc:4584] TS 3ef3213427b24f19a480330bb0fb4e9a (127.2.123.1:36267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca273b5e96c64d5795515784b207b1fd: Network error: Client connection negotiation failed: client connection to 127.2.123.1:36267: connect: Connection refused (error 111)
I20260928 02:43:53.318610 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 15637
I20260928 02:43:53.323626 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 15269
I20260928 02:43:53.329185 2540 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:43:53.663945 2540 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:43:53.664057 2540 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:43:53.998445 2540 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:43:53.998 UTC [14336] LOG: received smart shutdown request
2026-09-28 02:43:54.002 UTC [14336] LOG: background worker "logical replication launcher" (PID 14345) exited with exit code 1
2026-09-28 02:43:54.002 UTC [14340] LOG: shutting down
2026-09-28 02:43:54.163 UTC [14336] LOG: database system is shut down
2026-09-28T02:43:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48293 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:54 dist-test-slave-rqvb krb5kdc[15845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:54 dist-test-slave-rqvb krb5kdc[15845](info): set up 2 sockets
Sep 28 02:43:54 dist-test-slave-rqvb krb5kdc[15845](info): commencing operation
krb5kdc: starting...
W20260928 02:43:55.217841 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:43:55.333638 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:37283
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:37283,127.2.123.61:45377,127.2.123.60:35199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:55.442376 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:55.442659 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:55.442716 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:55.446728 15858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:55.446794 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:55.446820 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:55.446853 15858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:55.446882 15858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:55.452242 15858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:37283,127.2.123.61:45377,127.2.123.60:35199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:37283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:55.453298 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:55.454398 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:55.462615 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:55.462658 15858 server_base.cc:1034] running on GCE node
W20260928 02:43:55.462622 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:55.462667 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:55.463549 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:55.464655 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:55.465838 15858 hybrid_clock.cc:648] HybridClock initialized: now 1790563435465805 us; error 43 us; skew 500 ppm
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:55.468660 15858 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:43:55.469538 15858 webserver.cc:460] Webserver started at http://127.2.123.62:38883/ using document root <none> and password file <none>
I20260928 02:43:55.470156 15858 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:55.470232 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:55.470461 15858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:55.472441 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "fa7076912c2945189f1d388e10d23373"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "84745f6cfaf599c13769df35fcf777a8"
server_key_iv: "f01598bdfbc07d7b787e88d40aa293e0"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.472959 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "fa7076912c2945189f1d388e10d23373"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "84745f6cfaf599c13769df35fcf777a8"
server_key_iv: "f01598bdfbc07d7b787e88d40aa293e0"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.476604 15858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 02:43:55.479300 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:55.480656 15858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:43:55.480778 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "fa7076912c2945189f1d388e10d23373"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "84745f6cfaf599c13769df35fcf777a8"
server_key_iv: "f01598bdfbc07d7b787e88d40aa293e0"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.480882 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:55.504405 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:55.507720 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:55.507889 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:55.514371 15858 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:37283
I20260928 02:43:55.514381 15925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:37283 every 8 connection(s)
I20260928 02:43:55.515316 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:55.518287 15926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:55.519707 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 15858
I20260928 02:43:55.519824 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:43:55.520058 2540 external_mini_cluster.cc:1428] Setting key ae5e7546d0dfb3eb1d43f51fd6dd5d82
I20260928 02:43:55.522282 15926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.123.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.61@KRBTEST.COM" created.
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:55.537986 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.524623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37283 (local address 127.0.0.1:54234)
0928 02:43:55.525204 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.525260 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.526359 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.527109 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.527116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.527547 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.527926 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.527943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.529228 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.529230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.530102 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.530114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.531042 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.531045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.531143 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.531749 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.531770 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.533292 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.535134 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.535141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.535152 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.535361 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.535645 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.535648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.535650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.535755 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.536308 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.536314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.536446 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.536599 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.536974 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":88,"thread_start_us":143,"threads_started":1}
I20260928 02:43:55.537994 15930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.524623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.2.123.62:37283)
0928 02:43:55.525156 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.525172 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.525373 (+ 201us) server_negotiation.cc:407] Connection header received
0928 02:43:55.526702 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.526707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.526888 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.527052 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.528085 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.529118 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.530244 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.530822 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.531012 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.533425 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.533431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.533441 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.533460 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.535036 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.535458 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.535464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.535466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.535565 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.535831 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.535835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.535836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.536207 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.536322 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.536675 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.536891 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"spinlock_wait_cycles":50944,"thread_start_us":143,"threads_started":1}
I20260928 02:43:55.540344 15926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:55.541025 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.540777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42516)
0928 02:43:55.540887 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.540955 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:43:55.541224 15876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
I20260928 02:43:55.541996 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.541784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42518)
0928 02:43:55.541883 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.541924 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:43:55.542121 15926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:45377: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab.
Entry for principal kudu/127.2.123.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab.
I20260928 02:43:55.544893 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.61:45377
--webserver_interface=127.2.123.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab
--principal=kudu/127.2.123.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:37283,127.2.123.61:45377,127.2.123.60:35199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:55.591361 15926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } attempt: 1
I20260928 02:43:55.592183 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.591907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42530)
0928 02:43:55.592044 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.592107 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:43:55.592896 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.592672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42540)
0928 02:43:55.592756 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.592838 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 02:43:55.593021 15926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:45377: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
W20260928 02:43:55.653769 15933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:55.654060 15933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:55.654102 15933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:55.658038 15933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:55.658097 15933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:55.658113 15933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:55.658129 15933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:55.658142 15933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:55.661171 15926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } attempt: 2
I20260928 02:43:55.662019 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.661677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42550)
0928 02:43:55.661820 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.661904 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:43:55.662644 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.662445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42554)
0928 02:43:55.662538 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.662580 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:43:55.662744 15926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:45377: Network error: Client connection negotiation failed: client connection to 127.2.123.61:45377: connect: Connection refused (error 111)
I20260928 02:43:55.663729 15933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:37283,127.2.123.61:45377,127.2.123.60:35199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab
--principal=kudu/127.2.123.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.61:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.123.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:55.664770 15933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:55.665755 15933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:55.674115 15938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:55.674185 15941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:55.674089 15939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:55.674289 15933 server_base.cc:1034] running on GCE node
I20260928 02:43:55.674892 15933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:55.676016 15933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:55.677197 15933 hybrid_clock.cc:648] HybridClock initialized: now 1790563435677176 us; error 32 us; skew 500 ppm
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:55.679881 15933 init.cc:377] Logged in from keytab as kudu/127.2.123.61@KRBTEST.COM (short username kudu)
I20260928 02:43:55.680879 15933 webserver.cc:460] Webserver started at http://127.2.123.61:39961/ using document root <none> and password file <none>
I20260928 02:43:55.681418 15933 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:55.681483 15933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:55.681672 15933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:55.683822 15933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/data/instance:
uuid: "5ec54169cc33476fae6975a13b013c84"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "5e21d32d056cd430c2f679a187a9c8d5"
server_key_iv: "770504ffae3d69ef49aad5ac918533b6"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.684424 15933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/wal/instance:
uuid: "5ec54169cc33476fae6975a13b013c84"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "5e21d32d056cd430c2f679a187a9c8d5"
server_key_iv: "770504ffae3d69ef49aad5ac918533b6"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.688264 15933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:43:55.690790 15948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:55.692327 15933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:55.692442 15933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/wal
uuid: "5ec54169cc33476fae6975a13b013c84"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "5e21d32d056cd430c2f679a187a9c8d5"
server_key_iv: "770504ffae3d69ef49aad5ac918533b6"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.692584 15933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:55.706333 15933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:55.709779 15933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:55.710021 15933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:55.716745 15933 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.61:45377
I20260928 02:43:55.716786 16000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.61:45377 every 8 connection(s)
I20260928 02:43:55.717703 15933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
I20260928 02:43:55.720443 16001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:55.720844 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 15933
I20260928 02:43:55.720964 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-1/wal/instance
I20260928 02:43:55.721239 2540 external_mini_cluster.cc:1428] Setting key 740bf9072f46fe1ae8dc538bad83e2ff
I20260928 02:43:55.724632 16001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.123.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.60@KRBTEST.COM" created.
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:55.738715 15930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.726657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.2.123.62:37283)
0928 02:43:55.726778 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.726781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.727919 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 02:43:55.728893 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.728898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.728963 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.729037 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.730286 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.730905 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.732387 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.732648 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.732748 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.735347 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.735350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.735352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.735378 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.736685 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.737183 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.737186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.737187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.737263 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.737580 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.737583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.737585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.737765 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.737848 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.738420 (+ 572us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.738553 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:43:55.739549 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.727185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37283 (local address 127.0.0.1:54242)
0928 02:43:55.727755 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.727796 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.728669 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.729219 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.729227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.729606 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.730143 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.730159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.731040 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.731043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.732248 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.732259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.732807 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.732812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.733022 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.733569 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.733585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.735172 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.736798 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.736805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.736832 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.737086 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.737343 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.737346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.737348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.737489 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.737862 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.737869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.738177 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.738351 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.738697 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":178,"threads_started":1}
I20260928 02:43:55.741041 16001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab.
Entry for principal kudu/127.2.123.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab.
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:43:55.746932 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.60:35199
--webserver_interface=127.2.123.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab
--principal=kudu/127.2.123.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:37283,127.2.123.61:45377,127.2.123.60:35199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:55.750617 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.741585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42564)
0928 02:43:55.741754 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.741766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.741883 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.742422 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.742424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.742572 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.742669 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.742675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.743721 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.743724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.744324 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.744341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.744970 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.744974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.745050 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.745503 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.745515 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.746958 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.748886 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.748889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.748891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.749136 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.749431 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.749435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.749437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.749552 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.750037 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.750040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.750150 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.750268 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.750432 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 02:43:55.750751 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.741714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.2.123.61:45377)
0928 02:43:55.742017 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.742037 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.742062 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:43:55.742106 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.742110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.742264 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.742375 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.742780 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.743591 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.744456 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.744843 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.744945 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.747114 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.747126 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.747142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.747169 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.748769 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.749234 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.749239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.749240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.749310 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.749653 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.749655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.749657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.749911 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.750073 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.750414 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.750553 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260928 02:43:55.752985 16001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:55.753758 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.753431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42244)
0928 02:43:55.753571 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.753670 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:43:55.753901 15926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } attempt: 3
W20260928 02:43:55.754020 15951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
I20260928 02:43:55.756119 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.755917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42248)
0928 02:43:55.756021 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.756063 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:43:55.756340 16001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:35199: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:43:55.763628 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.754461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.2.123.61:45377)
0928 02:43:55.754575 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.754580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.754623 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:43:55.754741 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.754745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.754797 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.754894 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.755834 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.757028 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.757995 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.758382 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.758502 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.760570 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.760573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.760575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.760608 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.761999 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.762485 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.762488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.762489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.762586 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.762866 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.762870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.762872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.763055 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.763146 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.763428 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.763524 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:43:55.763635 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.754391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42566)
0928 02:43:55.754565 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.754579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.754662 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.754941 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.754944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.755178 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.755706 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.755715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.757168 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.757172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.757758 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.757769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.758531 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.758534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.758626 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.759140 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.759159 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.760444 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.762100 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.762104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.762108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.762375 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.762671 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.762673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.762675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.762762 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.763144 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.763147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.763250 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.763347 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.763478 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 02:43:55.764479 15926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:55.765144 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.764929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42264)
0928 02:43:55.765028 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.765081 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:43:55.765770 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.765559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42274)
0928 02:43:55.765672 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.765716 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:43:55.765908 15926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:35199: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
I20260928 02:43:55.797120 15926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } attempt: 1
I20260928 02:43:55.798110 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.797696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42284)
0928 02:43:55.797869 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.798028 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:43:55.798784 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.798584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42286)
0928 02:43:55.798680 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.798726 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:43:55.798931 15926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:35199: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
I20260928 02:43:55.805617 16001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } attempt: 1
I20260928 02:43:55.806414 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.806156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42288)
0928 02:43:55.806289 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.806343 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:43:55.807075 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.806866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42290)
0928 02:43:55.806972 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.807021 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:43:55.807179 16001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:35199: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
W20260928 02:43:55.860796 16007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:55.861070 16007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:55.861127 16007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:55.865097 16007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:55.865159 16007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:55.865183 16007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:55.865206 16007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:55.865235 16007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:55.870512 16007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:37283,127.2.123.61:45377,127.2.123.60:35199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab
--principal=kudu/127.2.123.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.60:35199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.123.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:55.871572 16007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:55.872576 16007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:43:55.877696 15926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } attempt: 2
I20260928 02:43:55.878859 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.878466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42300)
0928 02:43:55.878699 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.878765 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 02:43:55.881844 16001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } attempt: 2
I20260928 02:43:55.882387 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.882108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42306)
0928 02:43:55.882265 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.882315 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 02:43:55.882558 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.882269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42308)
0928 02:43:55.882411 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.882471 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:43:55.882690 15926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:35199: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
I20260928 02:43:55.883363 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.883086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42320)
0928 02:43:55.883244 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.883298 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260928 02:43:55.883523 16001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:35199: Network error: Client connection negotiation failed: client connection to 127.2.123.60:35199: connect: Connection refused (error 111)
W20260928 02:43:55.886585 16013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:55.886617 16016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:55.886583 16014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:55.887037 16007 server_base.cc:1034] running on GCE node
I20260928 02:43:55.887565 16007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:55.888638 16007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:55.889796 16007 hybrid_clock.cc:648] HybridClock initialized: now 1790563435889770 us; error 38 us; skew 500 ppm
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:55.892652 16007 init.cc:377] Logged in from keytab as kudu/127.2.123.60@KRBTEST.COM (short username kudu)
I20260928 02:43:55.893537 16007 webserver.cc:460] Webserver started at http://127.2.123.60:44665/ using document root <none> and password file <none>
I20260928 02:43:55.894099 16007 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:55.894169 16007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:55.894363 16007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:55.896363 16007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/data/instance:
uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "ccea8f6270677cde307439b2a8b497e4"
server_key_iv: "31c64b97691817112231c2e354ea7fae"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.896908 16007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/wal/instance:
uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "ccea8f6270677cde307439b2a8b497e4"
server_key_iv: "31c64b97691817112231c2e354ea7fae"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.900472 16007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:43:55.902796 16023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:55.903959 16007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:43:55.904074 16007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/wal
uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd"
format_stamp: "Formatted at 2026-09-28 02:43:55 on dist-test-slave-rqvb"
server_key: "ccea8f6270677cde307439b2a8b497e4"
server_key_iv: "31c64b97691817112231c2e354ea7fae"
server_key_version: "encryptionkey@0"
I20260928 02:43:55.904160 16007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:55.921499 16007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:55.924520 16007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:55.924700 16007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:55.931432 16007 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.60:35199
I20260928 02:43:55.931495 16075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.60:35199 every 8 connection(s)
I20260928 02:43:55.932374 16007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
I20260928 02:43:55.934129 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 16007
I20260928 02:43:55.934271 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563278472902-2540-0/minicluster-data/master-2/wal/instance
I20260928 02:43:55.934617 2540 external_mini_cluster.cc:1428] Setting key e6c0a5485a4d56f41a5e1398829ebdce
I20260928 02:43:55.935508 16076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:55.939332 16076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:55.948333 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.935986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.2.123.60:35199)
0928 02:43:55.936403 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.936418 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.936454 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:43:55.937109 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.937126 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.937439 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.937741 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.938560 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.939521 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.940360 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.940972 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.941126 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.943666 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.943671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.943680 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.943704 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.945376 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.945771 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.945776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.945780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.945880 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.946148 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.946151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.946153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.946510 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.946697 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.947139 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.947384 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":134,"threads_started":1}
I20260928 02:43:55.949608 15930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.941040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.2.123.62:37283)
0928 02:43:55.941177 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.941182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.941584 (+ 402us) server_negotiation.cc:407] Connection header received
0928 02:43:55.941887 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.941891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.941949 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.942049 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.942738 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.943302 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.944206 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.944421 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.944508 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.946532 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.946536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.946538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.946564 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.948004 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.948411 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.948415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.948416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.948509 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.948783 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.948787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.948788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.948962 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.949051 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.949397 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.949494 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:43:55.949648 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.940923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37283 (local address 127.0.0.1:54246)
0928 02:43:55.941463 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.941503 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.941623 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.942115 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.942121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.942473 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.942616 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.942623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.943420 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.943423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.944095 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.944104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.944516 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.944519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.944620 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.945028 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.945039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.946390 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.948109 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.948117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.948121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.948318 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.948613 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.948617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.948619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.948699 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.949067 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.949073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.949174 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.949321 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.949496 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":143,"threads_started":1}
I20260928 02:43:55.950655 16076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:43:55.958756 15930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.950633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.2.123.62:37283)
0928 02:43:55.950739 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.950742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.950821 (+ 79us) server_negotiation.cc:407] Connection header received
0928 02:43:55.950922 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.950925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.950967 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.951030 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.951638 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.952147 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.952832 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.953148 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.953238 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.955686 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.955690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.955693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.955718 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.957220 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.957573 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.957577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.957579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.957660 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.957916 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.957919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.957920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.958139 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.958352 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.958491 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.958607 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:43:55.959180 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.951126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:45377 (local address 127.0.0.1:42570)
0928 02:43:55.951345 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.951357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.951429 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.952257 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.952259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.952375 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.952455 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.952459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.953333 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.953336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.954086 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.954098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.954516 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.954519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.954627 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.955014 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.955023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.956293 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.957774 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.957777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.957779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.958024 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.958274 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.958276 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.958278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.958364 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.958747 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.958750 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.958827 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.958923 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.959037 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 02:43:55.959249 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.951233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.2.123.61:45377)
0928 02:43:55.951344 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.951348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.951396 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:43:55.951503 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.951506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.951549 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.951620 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.952580 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.953232 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.954201 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.954405 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.954485 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.956427 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.956430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.956433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.956462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.957685 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.958110 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.958114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.958116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.958196 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.958449 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.958452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.958454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.958649 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.958739 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.958981 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.959099 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:55.959934 16076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:43:55.967720 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.960440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42344)
0928 02:43:55.960525 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.960538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.960609 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.960848 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.960851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.960997 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.961079 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.961084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.962084 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.962087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.962570 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.962577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.963069 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.963073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.963157 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.963484 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.963498 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.964731 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.966395 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.966398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.966401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.966580 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.966807 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.966810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.966812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.966886 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.967243 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.967246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.967329 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.967427 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.967584 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:43:55.967833 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.960489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.2.123.60:35199)
0928 02:43:55.960703 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.960706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.960713 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:55.960731 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.960733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.960757 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.960802 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.961176 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.962001 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.962663 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.962945 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.963040 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.964860 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.964864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.964866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.964886 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.966296 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.966664 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.966667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.966669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.966734 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.966963 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.966966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.966967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.967149 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.967242 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.967478 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.967658 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":170}
I20260928 02:43:55.968633 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.960040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.2.123.61:45377)
0928 02:43:55.960143 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.960146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.960159 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:43:55.960277 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.960280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.960320 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.960391 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.960970 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.961487 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.962258 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.962466 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.962546 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.965426 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.965430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.965432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.965454 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.966826 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.967220 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.967223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.967225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.967309 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.967577 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.967580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.967582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.967787 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.968032 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.968367 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.968475 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:43:55.971063 16076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd: Bootstrap starting.
I20260928 02:43:55.972419 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:55.973937 16076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:55.974764 16001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } attempt: 3
I20260928 02:43:55.974923 16076 log.cc:824] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:43:55.982232 16076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd: No bootstrap required, opened a new log
I20260928 02:43:55.982633 15926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } attempt: 3
I20260928 02:43:55.984635 15930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.975028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.2.123.62:37283)
0928 02:43:55.975554 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.975558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.975572 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:43:55.975613 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.975616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.975659 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.975723 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.976490 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.977219 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.978481 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.978837 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.978935 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.980754 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.980759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.980762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.980793 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.982469 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.982893 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.982897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.982899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.982981 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.983397 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.983400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.983402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.983611 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.983726 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.984364 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.984478 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441}
I20260928 02:43:55.985599 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.975161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.2.123.60:35199)
0928 02:43:55.975394 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.975398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.975593 (+ 195us) server_negotiation.cc:407] Connection header received
0928 02:43:55.975687 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.975690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.975732 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.975803 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.976559 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.977263 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.978449 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.978775 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.978852 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.981296 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.981300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.981303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.981326 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.983021 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.983571 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.983574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.983576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.983874 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.984450 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.984453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.984455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.984661 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.984754 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.985151 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.985464 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260928 02:43:55.986963 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.975169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.2.123.61:45377)
0928 02:43:55.976042 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.976046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.976059 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:43:55.976093 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.976097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.976140 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.976214 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.977033 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.978066 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.979376 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.979706 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.979799 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.983430 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.983433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.983436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.983459 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.984910 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.985371 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.985375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.985377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.985452 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.985726 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.985730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.985732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.985944 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.986102 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.986675 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.986786 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":801}
I20260928 02:43:55.988152 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.975373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42362)
0928 02:43:55.975470 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.975483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.975560 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.976813 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.976816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.976993 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.977127 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.977133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.978109 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.978112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.978885 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.978895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.979490 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.979493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.979584 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.980114 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.980130 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.981646 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.986786 (+ 5140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.986789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.986791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.986994 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.987292 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.987295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.987297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.987374 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.987698 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.987701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.987786 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.987877 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.987999 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":19}
I20260928 02:43:55.988687 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.976022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.2.123.60:35199)
0928 02:43:55.976593 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.976601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.976612 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:55.976641 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.976644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.976680 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.977071 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.977204 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.977988 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.979014 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.979342 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.979483 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.984978 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.984984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.984988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.985013 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.986665 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.987111 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.987114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.987116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.987193 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.987453 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.987456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.987458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.987612 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.987692 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.988421 (+ 729us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.988529 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":114,"threads_started":1}
I20260928 02:43:55.989065 16076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:55.989421 16076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:55.989519 16076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9901a4a96a4cf2a8f7cc5f765444bd, State: Initialized, Role: FOLLOWER
I20260928 02:43:55.990034 16076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:55.990496 16091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:55.990688 16091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 02:43:55 dist-test-slave-rqvb krb5kdc[15845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:43:55.991379 16076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:55.991983 16001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84: Bootstrap starting.
I20260928 02:43:55.994901 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.983045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:35199 (local address 127.0.0.1:42372)
0928 02:43:55.983224 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.983238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:55.983313 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:55.986454 (+ 3141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:55.986457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:55.986630 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:55.986733 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.986739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.987881 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.987884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.988631 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:55.988641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.989124 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.989128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:55.989208 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.990061 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:55.990075 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:55.991453 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:55.993315 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.993318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.993320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.993552 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.993840 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:55.993856 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:55.993859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:55.993945 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:55.994431 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:55.994434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:55.994528 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 02:43:55.994621 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:43:55.994744 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 02:43:55.995385 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.983091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.2.123.60:35199)
0928 02:43:55.986245 (+ 3154us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:55.986249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:55.986261 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:43:55.986291 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:55.986293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:55.986332 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:55.986837 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:55.986915 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.987778 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.988757 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:55.989021 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:55.989107 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:55.991570 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:55.991574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:55.991577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:55.991599 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:55.993214 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.993660 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.993664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.993665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.993743 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:55.994083 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:55.994086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:55.994088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:55.994305 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:55.994387 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:55.995092 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 02:43:55.995194 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3099,"thread_start_us":105,"threads_started":1}
I20260928 02:43:55.996388 16001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84: Neither blocks nor log segments found. Creating new log.
W20260928 02:43:55.997231 16102 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:55.997329 16102 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:55.998356 15926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373: Bootstrap starting.
I20260928 02:43:55.998545 16001 log.cc:824] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:55.995218 16007 master_runner.cc:186] Error getting master registration for 127.2.123.62:37283: OK, instance_id { permanent_uuid: "fa7076912c2945189f1d388e10d23373" instance_seqno: 1790563435512580 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:43:55.998929 16007 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:43:56.000835 16001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84: No bootstrap required, opened a new log
I20260928 02:43:56.001014 15926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:56.001821 15926 log.cc:824] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:56.003863 15926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373: No bootstrap required, opened a new log
I20260928 02:43:56.004580 16001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:56.006044 16001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:56.006129 16001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec54169cc33476fae6975a13b013c84, State: Initialized, Role: FOLLOWER
I20260928 02:43:56.006603 16001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:56.007452 15926 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:56.007616 15926 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:56.007694 15926 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7076912c2945189f1d388e10d23373, State: Initialized, Role: FOLLOWER
I20260928 02:43:56.007979 16001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:56.008123 15926 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:56.008561 16107 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.008728 16107 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:56.009289 15926 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:43:56.009392 15879 tablet.cc:2369] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:43:56.010218 16106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.010469 16106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:43:56.013073 16119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:56.013195 16119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 02:43:56.014655 16129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:56.014776 16129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:56.011262 15933 master_runner.cc:186] Error getting master registration for 127.2.123.62:37283: OK, instance_id { permanent_uuid: "fa7076912c2945189f1d388e10d23373" instance_seqno: 1790563435512580 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:43:56.016597 15933 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:43:56.020407 15858 master_runner.cc:263] No leader master found from master fa7076912c2945189f1d388e10d23373
I20260928 02:43:56.020483 15858 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:43:56.059815 16107 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:56.060119 16107 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:56.061280 16107 leader_election.cc:288] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec54169cc33476fae6975a13b013c84 (127.2.123.61:45377), 4a9901a4a96a4cf2a8f7cc5f765444bd (127.2.123.60:35199)
I20260928 02:43:56.062292 15975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7076912c2945189f1d388e10d23373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec54169cc33476fae6975a13b013c84" is_pre_election: true
I20260928 02:43:56.062646 15975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7076912c2945189f1d388e10d23373 in term 0.
I20260928 02:43:56.063638 16050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7076912c2945189f1d388e10d23373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" is_pre_election: true
I20260928 02:43:56.063961 16050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7076912c2945189f1d388e10d23373 in term 0.
I20260928 02:43:56.065212 15876 leader_election.cc:302] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9901a4a96a4cf2a8f7cc5f765444bd, fa7076912c2945189f1d388e10d23373; no voters:
I20260928 02:43:56.066524 16107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:56.066620 16107 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:56.066664 16107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:56.068008 16107 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:56.068365 16107 leader_election.cc:288] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [CANDIDATE]: Term 1 election: Requested vote from peers 5ec54169cc33476fae6975a13b013c84 (127.2.123.61:45377), 4a9901a4a96a4cf2a8f7cc5f765444bd (127.2.123.60:35199)
I20260928 02:43:56.068961 16050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7076912c2945189f1d388e10d23373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd"
I20260928 02:43:56.069082 16050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:56.070355 16050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7076912c2945189f1d388e10d23373 in term 1.
I20260928 02:43:56.070796 15975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7076912c2945189f1d388e10d23373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec54169cc33476fae6975a13b013c84"
I20260928 02:43:56.070922 15975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:56.072167 15975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7076912c2945189f1d388e10d23373 in term 1.
I20260928 02:43:56.072692 15876 leader_election.cc:302] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec54169cc33476fae6975a13b013c84, fa7076912c2945189f1d388e10d23373; no voters:
I20260928 02:43:56.073678 16107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:56.073994 16107 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [term 1 LEADER]: Becoming Leader. State: Replica: fa7076912c2945189f1d388e10d23373, State: Running, Role: LEADER
I20260928 02:43:56.074445 16107 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } }
I20260928 02:43:56.076357 16107 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7076912c2945189f1d388e10d23373. Latest consensus state: current_term: 1 leader_uuid: "fa7076912c2945189f1d388e10d23373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.076547 16107 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:56.076974 16134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:56.078197 16134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:56.086298 16050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [term 1 FOLLOWER]: Refusing update from remote peer fa7076912c2945189f1d388e10d23373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:56.088737 16107 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:56.093772 16091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7076912c2945189f1d388e10d23373. Latest consensus state: current_term: 1 leader_uuid: "fa7076912c2945189f1d388e10d23373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.093912 16091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:56.095140 15975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [term 1 FOLLOWER]: Refusing update from remote peer fa7076912c2945189f1d388e10d23373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:56.095827 16107 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:56.100286 16106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7076912c2945189f1d388e10d23373. Latest consensus state: current_term: 1 leader_uuid: "fa7076912c2945189f1d388e10d23373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.100417 16106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:56.107888 16107 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa7076912c2945189f1d388e10d23373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.108047 16107 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:56.109344 16106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa7076912c2945189f1d388e10d23373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.109473 16106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:56.110035 16135 mvcc.cc:205] Tried to move back new op lower bound from 7334147834197958656 to 7334147834164568064. Current Snapshot: MvccSnapshot[applied={T|T < 7334147834197958656}]
I20260928 02:43:56.111706 16138 mvcc.cc:205] Tried to move back new op lower bound from 7334147834197958656 to 7334147834164568064. Current Snapshot: MvccSnapshot[applied={T|T < 7334147834197958656}]
I20260928 02:43:56.116375 16132 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa7076912c2945189f1d388e10d23373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.116528 16132 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:56.117879 16091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa7076912c2945189f1d388e10d23373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7076912c2945189f1d388e10d23373" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37283 } } peers { permanent_uuid: "5ec54169cc33476fae6975a13b013c84" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 45377 } } peers { permanent_uuid: "4a9901a4a96a4cf2a8f7cc5f765444bd" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 35199 } } }
I20260928 02:43:56.118028 16091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:56.122925 16136 mvcc.cc:205] Tried to move back new op lower bound from 7334147834197958656 to 7334147834164568064. Current Snapshot: MvccSnapshot[applied={T|T < 7334147834197958656}]
I20260928 02:43:56.123574 16134 catalog_manager.cc:1347] Generated new cluster ID: 7f868fb92f6447c6a940ede3a4612876
I20260928 02:43:56.123636 16134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:56.160523 16134 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:56.161840 16134 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:56.185024 16134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa7076912c2945189f1d388e10d23373: Generated new TSK 0
I20260928 02:43:56.186096 16134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:56.998636 16102 catalog_manager.cc:1259] Loaded cluster ID: 7f868fb92f6447c6a940ede3a4612876
I20260928 02:43:56.998718 16102 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd: loading cluster ID for follower catalog manager: success
I20260928 02:43:57.000319 16102 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd: acquiring CA information for follower catalog manager: success
I20260928 02:43:57.001451 16102 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a9901a4a96a4cf2a8f7cc5f765444bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:43:57.014508 16119 catalog_manager.cc:1259] Loaded cluster ID: 7f868fb92f6447c6a940ede3a4612876
I20260928 02:43:57.014580 16119 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84: loading cluster ID for follower catalog manager: success
I20260928 02:43:57.016196 16119 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84: acquiring CA information for follower catalog manager: success
I20260928 02:43:57.017277 16119 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ec54169cc33476fae6975a13b013c84: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:43:57.255113 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 15858
I20260928 02:43:57.260663 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 15933
I20260928 02:43:57.265414 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 16007
2026-09-28T02:43:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3007 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:57 dist-test-slave-rqvb krb5kdc[16148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:57 dist-test-slave-rqvb krb5kdc[16148](info): set up 2 sockets
Sep 28 02:43:57 dist-test-slave-rqvb krb5kdc[16148](info): commencing operation
krb5kdc: starting...
W20260928 02:43:58.222044 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:43:58 dist-test-slave-rqvb krb5kdc[16148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:43:58.341281 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:36553
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:36553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:58.449204 16161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:58.449491 16161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:58.449548 16161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:58.453684 16161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:58.453779 16161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:58.453807 16161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:58.453830 16161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:58.453850 16161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:58.459486 16161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:36553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:58.460645 16161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:58.461669 16161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:58.470252 16166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:58.470312 16161 server_base.cc:1034] running on GCE node
W20260928 02:43:58.470261 16167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:58.470276 16169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:58.471127 16161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:58.472237 16161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:58.473399 16161 hybrid_clock.cc:648] HybridClock initialized: now 1790563438473373 us; error 42 us; skew 500 ppm
Sep 28 02:43:58 dist-test-slave-rqvb krb5kdc[16148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563438, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:58.476105 16161 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:43:58.476970 16161 webserver.cc:460] Webserver started at http://127.2.123.62:40651/ using document root <none> and password file <none>
I20260928 02:43:58.477488 16161 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:58.477546 16161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:58.477700 16161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:58.479708 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "e59da4c1149d443d91373890e686a55f"
format_stamp: "Formatted at 2026-09-28 02:43:58 on dist-test-slave-rqvb"
server_key: "7095b5d00bbd72ad1ae97f22c9cdab83"
server_key_iv: "681d8be397201f80f81b9b08d944a884"
server_key_version: "encryptionkey@0"
I20260928 02:43:58.480212 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "e59da4c1149d443d91373890e686a55f"
format_stamp: "Formatted at 2026-09-28 02:43:58 on dist-test-slave-rqvb"
server_key: "7095b5d00bbd72ad1ae97f22c9cdab83"
server_key_iv: "681d8be397201f80f81b9b08d944a884"
server_key_version: "encryptionkey@0"
I20260928 02:43:58.483808 16161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:43:58.486284 16176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:58.487568 16161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:43:58.487701 16161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "e59da4c1149d443d91373890e686a55f"
format_stamp: "Formatted at 2026-09-28 02:43:58 on dist-test-slave-rqvb"
server_key: "7095b5d00bbd72ad1ae97f22c9cdab83"
server_key_iv: "681d8be397201f80f81b9b08d944a884"
server_key_version: "encryptionkey@0"
I20260928 02:43:58.487800 16161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:58.508490 16161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:58.511577 16161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:58.511806 16161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:58.519002 16161 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:36553
I20260928 02:43:58.519047 16228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:36553 every 8 connection(s)
I20260928 02:43:58.519946 16161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:58.522774 16229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:58.527623 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 16161
I20260928 02:43:58.527745 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:43:58.528007 2540 external_mini_cluster.cc:1428] Setting key 5abf9ffa2197588730c35508e3e781a9
I20260928 02:43:58.528359 16229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f: Bootstrap starting.
I20260928 02:43:58.530900 16229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:58.531625 16229 log.cc:824] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:58.533419 16229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f: No bootstrap required, opened a new log
Sep 28 02:43:58 dist-test-slave-rqvb krb5kdc[16148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:58.537262 16229 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59da4c1149d443d91373890e686a55f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36553 } }
I20260928 02:43:58.537494 16229 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:58.537533 16229 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59da4c1149d443d91373890e686a55f, State: Initialized, Role: FOLLOWER
I20260928 02:43:58.538075 16229 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59da4c1149d443d91373890e686a55f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36553 } }
I20260928 02:43:58.538245 16229 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:58.538318 16229 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:58.538421 16229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:58.539318 16229 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59da4c1149d443d91373890e686a55f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36553 } }
I20260928 02:43:58.539695 16229 leader_election.cc:302] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59da4c1149d443d91373890e686a55f; no voters:
I20260928 02:43:58.540074 16229 leader_election.cc:288] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:58.540155 16234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:58.540380 16234 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [term 1 LEADER]: Becoming Leader. State: Replica: e59da4c1149d443d91373890e686a55f, State: Running, Role: LEADER
I20260928 02:43:58.540747 16234 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59da4c1149d443d91373890e686a55f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36553 } }
I20260928 02:43:58.541447 16229 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:58.542616 16236 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59da4c1149d443d91373890e686a55f. Latest consensus state: current_term: 1 leader_uuid: "e59da4c1149d443d91373890e686a55f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59da4c1149d443d91373890e686a55f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36553 } } }
I20260928 02:43:58.542742 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:58.529419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.2.123.62:36553)
0928 02:43:58.529921 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:58.529936 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:58.529994 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:43:58.530675 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:58.530694 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:58.531041 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:58.531374 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:58.532180 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:58.533160 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:58.534056 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:58.534529 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:58.534684 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:58.537259 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:58.537264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:58.537278 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:58.537306 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:58.539073 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:58.539564 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:58.539570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:58.539577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:58.539705 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:58.540087 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:58.540091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:58.540094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:58.540760 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:58.540982 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:58.541282 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 02:43:58.541658 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":179,"threads_started":1}
I20260928 02:43:58.542778 16236 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:58.542627 16235 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59da4c1149d443d91373890e686a55f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59da4c1149d443d91373890e686a55f" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 36553 } } }
I20260928 02:43:58.543874 16235 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:58.545779 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 02:43:58.546608 16249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:58.546684 16249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:58.546811 16243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:58.547535 16243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:58.553514 16243 catalog_manager.cc:1347] Generated new cluster ID: a4d36326159e45ae89eb615458e1d049
I20260928 02:43:58.553602 16243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:58.586345 16243 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:58.587700 16243 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:58.604640 16243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59da4c1149d443d91373890e686a55f: Generated new TSK 0
I20260928 02:43:58.605561 16243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:58.652978 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 16161
2026-09-28T02:43:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1386 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:58 dist-test-slave-rqvb krb5kdc[16256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:58 dist-test-slave-rqvb krb5kdc[16256](info): set up 2 sockets
Sep 28 02:43:58 dist-test-slave-rqvb krb5kdc[16256](info): commencing operation
krb5kdc: starting...
W20260928 02:43:59.601882 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:43:59 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:43:59.715274 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40621
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:59.821699 16269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:59.821995 16269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:59.822054 16269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:59.825948 16269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:59.826036 16269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:59.826062 16269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:59.826081 16269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:59.826102 16269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:59.831713 16269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:43:59.832903 16269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:59.833990 16269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:59.842535 16277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:59.842559 16275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:59.842535 16274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:59.842816 16269 server_base.cc:1034] running on GCE node
I20260928 02:43:59.843425 16269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:59.844565 16269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:59.845801 16269 hybrid_clock.cc:648] HybridClock initialized: now 1790563439845786 us; error 30 us; skew 500 ppm
Sep 28 02:43:59 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:59.848518 16269 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:43:59.849386 16269 webserver.cc:460] Webserver started at http://127.2.123.62:38225/ using document root <none> and password file <none>
I20260928 02:43:59.849900 16269 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:59.849954 16269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:59.850162 16269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:59.852119 16269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "692fc8a19f074f9c83a0197417fe5a96"
format_stamp: "Formatted at 2026-09-28 02:43:59 on dist-test-slave-rqvb"
server_key: "8b2b8d922f1e24ea8acdcb274fb64909"
server_key_iv: "6172a777a47a3e70bd42229c47cc104a"
server_key_version: "encryptionkey@0"
I20260928 02:43:59.852610 16269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "692fc8a19f074f9c83a0197417fe5a96"
format_stamp: "Formatted at 2026-09-28 02:43:59 on dist-test-slave-rqvb"
server_key: "8b2b8d922f1e24ea8acdcb274fb64909"
server_key_iv: "6172a777a47a3e70bd42229c47cc104a"
server_key_version: "encryptionkey@0"
I20260928 02:43:59.856455 16269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:43:59.858886 16284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:59.860158 16269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:43:59.860265 16269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "692fc8a19f074f9c83a0197417fe5a96"
format_stamp: "Formatted at 2026-09-28 02:43:59 on dist-test-slave-rqvb"
server_key: "8b2b8d922f1e24ea8acdcb274fb64909"
server_key_iv: "6172a777a47a3e70bd42229c47cc104a"
server_key_version: "encryptionkey@0"
I20260928 02:43:59.860351 16269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:59.874522 16269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:59.878216 16269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:59.878410 16269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:59.885206 16269 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:40621
I20260928 02:43:59.885524 16336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:40621 every 8 connection(s)
I20260928 02:43:59.886201 16269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:59.888952 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:59.891194 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 16269
I20260928 02:43:59.891311 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:43:59.891570 2540 external_mini_cluster.cc:1428] Setting key a101a7b805340ec0a0e7e10d659c6323
I20260928 02:43:59.895166 16337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Bootstrap starting.
I20260928 02:43:59.897755 16337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:59.898530 16337 log.cc:824] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:43:59 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:43:59.900586 16337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: No bootstrap required, opened a new log
I20260928 02:43:59.904202 16337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:43:59.904486 16337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:59.904554 16337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692fc8a19f074f9c83a0197417fe5a96, State: Initialized, Role: FOLLOWER
I20260928 02:43:59.904971 16337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:43:59.905107 16337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:59.905175 16337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:59.905279 16337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:59.905378 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:59.892894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.2.123.62:40621)
0928 02:43:59.893323 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:59.893349 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:59.893392 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:43:59.894220 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:59.894243 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:59.894640 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:59.895006 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:59.895661 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:59.897111 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:59.897984 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:59.898449 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:59.898598 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:59.900946 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:59.900951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:59.900963 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:59.900990 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:59.902606 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:59.903003 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:59.903015 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:59.903020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:59.903139 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:59.903394 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:59.903398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:59.903399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:59.903784 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:59.903964 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:59.904246 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:43:59.904538 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":122,"threads_started":1}
I20260928 02:43:59.906282 16337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:43:59.906615 16337 leader_election.cc:302] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692fc8a19f074f9c83a0197417fe5a96; no voters:
I20260928 02:43:59.906916 16337 leader_election.cc:288] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:59.907011 16342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:59.907241 16342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 1 LEADER]: Becoming Leader. State: Replica: 692fc8a19f074f9c83a0197417fe5a96, State: Running, Role: LEADER
I20260928 02:43:59.907569 16342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:43:59.908195 16337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:59.909026 16344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692fc8a19f074f9c83a0197417fe5a96. Latest consensus state: current_term: 1 leader_uuid: "692fc8a19f074f9c83a0197417fe5a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } } }
I20260928 02:43:59.909162 16344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:59.909502 16351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:59.910931 16343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "692fc8a19f074f9c83a0197417fe5a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } } }
I20260928 02:43:59.911038 16343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:59.912724 16351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:59.917752 16351 catalog_manager.cc:1347] Generated new cluster ID: 688465d0ea3649919a5ab57e6753a91e
I20260928 02:43:59.917831 16351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:59.932063 16351 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:59.933018 16351 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:59.938138 16351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Generated new TSK 0
I20260928 02:43:59.938776 16351 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:43:59.972993 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40621
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:00.081820 16363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:00.082127 16363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:00.082173 16363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:00.086153 16363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:00.086212 16363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:00.086273 16363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:44:00.091490 16363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:00.092555 16363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:00.093576 16363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:00.103128 16369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:00.103145 16368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:00.103192 16363 server_base.cc:1034] running on GCE node
W20260928 02:44:00.103142 16371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:00.103755 16363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:00.104391 16363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:00.105543 16363 hybrid_clock.cc:648] HybridClock initialized: now 1790563440105513 us; error 42 us; skew 500 ppm
Sep 28 02:44:00 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:00.108246 16363 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:44:00.109151 16363 webserver.cc:460] Webserver started at http://127.2.123.1:41447/ using document root <none> and password file <none>
I20260928 02:44:00.109750 16363 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:00.109814 16363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:00.110061 16363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:00.112144 16363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "349efd8e11884710b85bbb42e632add1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "d87e645cd2d04af52731449bf31f819e"
server_key_iv: "5e59f9a89d4ec6dfc3b4d8df558d5e0c"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.112685 16363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "349efd8e11884710b85bbb42e632add1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "d87e645cd2d04af52731449bf31f819e"
server_key_iv: "5e59f9a89d4ec6dfc3b4d8df558d5e0c"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.116386 16363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 02:44:00.118745 16378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.119969 16363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:44:00.120110 16363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "349efd8e11884710b85bbb42e632add1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "d87e645cd2d04af52731449bf31f819e"
server_key_iv: "5e59f9a89d4ec6dfc3b4d8df558d5e0c"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.120206 16363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:00.155881 16363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:00.158902 16363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:00.159088 16363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:00.159682 16363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:00.160566 16363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:00.160615 16363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.160656 16363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:00.160670 16363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.169405 16363 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:34991
I20260928 02:44:00.169425 16491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:34991 every 8 connection(s)
I20260928 02:44:00.170331 16363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:00.179428 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 16363
I20260928 02:44:00.179576 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
Sep 28 02:44:00 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:00.179893 2540 external_mini_cluster.cc:1428] Setting key f2544e76f8fa60df0d1b6eb1d935abb4
I20260928 02:44:00.184188 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.172057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:35843 (local address 127.2.123.62:40621)
0928 02:44:00.172220 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.172225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.173074 (+ 849us) server_negotiation.cc:407] Connection header received
0928 02:44:00.174318 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.174322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.174373 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.174464 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:00.175642 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.176201 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.177365 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.177613 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.177703 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.180449 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:00.180453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:00.180456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:00.180486 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:00.182012 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:00.182596 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:00.182599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:00.182600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:00.182679 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:00.183071 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:00.183074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:00.183076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:00.183305 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:00.183398 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.183889 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.184031 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:44:00.184979 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.172473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:35843)
0928 02:44:00.172904 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.172949 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:00.174100 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:00.174624 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:00.174631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:00.175011 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:00.175496 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.175510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.176322 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.176325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.177231 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.177240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.177740 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.177742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.177885 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.178551 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:00.178568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:00.180260 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:00.182123 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:00.182129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:00.182146 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:00.182480 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:00.182756 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:00.182759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:00.182761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:00.182958 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:00.183406 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:00.183411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:00.183667 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 02:44:00.183822 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:44:00.184085 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":97,"threads_started":1}
I20260928 02:44:00.186219 16492 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40621
I20260928 02:44:00.186498 16492 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:00.187086 16492 heartbeater.cc:511] Master 127.2.123.62:40621 requested a full tablet report, sending...
I20260928 02:44:00.188637 16297 ts_manager.cc:194] Registered new tserver with Master: 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991)
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:44:00.189776 16297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:35843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:44:00.205273 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40621
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:00.314214 16497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:00.314487 16497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:00.314532 16497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:00.318565 16497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:00.318624 16497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:00.318682 16497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:44:00.323920 16497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:00.325093 16497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:00.326220 16497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:00.335916 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:00.335906 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:00.336066 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:00.335986 16497 server_base.cc:1034] running on GCE node
I20260928 02:44:00.336498 16497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:00.337121 16497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:00.338271 16497 hybrid_clock.cc:648] HybridClock initialized: now 1790563440338277 us; error 67 us; skew 500 ppm
Sep 28 02:44:00 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:00.341053 16497 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:44:00.341933 16497 webserver.cc:460] Webserver started at http://127.2.123.2:41371/ using document root <none> and password file <none>
I20260928 02:44:00.342484 16497 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:00.342551 16497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:00.342743 16497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:00.344708 16497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "c325a449b4cf4e8c8373333962c734c4"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "b71aa98ef4115a4722ae5244e426a8f0"
server_key_iv: "70d348c4e578c39e9f5863812af8ff72"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.345222 16497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "c325a449b4cf4e8c8373333962c734c4"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "b71aa98ef4115a4722ae5244e426a8f0"
server_key_iv: "70d348c4e578c39e9f5863812af8ff72"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.348809 16497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 02:44:00.351203 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.352632 16497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:44:00.352730 16497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "c325a449b4cf4e8c8373333962c734c4"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "b71aa98ef4115a4722ae5244e426a8f0"
server_key_iv: "70d348c4e578c39e9f5863812af8ff72"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.352839 16497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:00.370432 16497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:00.373646 16497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:00.373817 16497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:00.374451 16497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:00.375276 16497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:00.375322 16497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.375362 16497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:00.375391 16497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.383767 16497 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:41921
I20260928 02:44:00.383836 16625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:41921 every 8 connection(s)
I20260928 02:44:00.384689 16497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:00.391414 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 16497
I20260928 02:44:00.391559 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:44:00.391845 2540 external_mini_cluster.cc:1428] Setting key 9d3083a4de3b706d0884786ece0c82da
Sep 28 02:44:00 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:00.398145 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.386465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:33681 (local address 127.2.123.62:40621)
0928 02:44:00.386627 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.386631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.387415 (+ 784us) server_negotiation.cc:407] Connection header received
0928 02:44:00.388612 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.388614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.388660 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.388732 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:00.389878 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.390518 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.391772 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.392087 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.392196 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.394924 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:00.394927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:00.394929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:00.394953 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:00.396231 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:00.396707 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:00.396711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:00.396712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:00.396788 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:00.397098 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:00.397101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:00.397103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:00.397278 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:00.397363 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.397839 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.397986 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:44:00.398967 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.386843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:33681)
0928 02:44:00.387267 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.387310 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:00.388418 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:00.388883 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:00.388889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:00.389245 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:00.389764 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.389777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.390646 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.390649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.391654 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.391665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.392245 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.392250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.392447 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.393029 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:00.393050 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:00.394763 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:00.396333 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:00.396338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:00.396350 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:00.396610 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:00.396870 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:00.396873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:00.396875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:00.397011 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:00.397375 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:00.397381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:00.397616 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 02:44:00.397779 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 02:44:00.398135 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260928 02:44:00.400285 16626 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40621
I20260928 02:44:00.400555 16626 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:00.401031 16626 heartbeater.cc:511] Master 127.2.123.62:40621 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:44:00.402086 16297 ts_manager.cc:194] Registered new tserver with Master: c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921)
I20260928 02:44:00.402631 16297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:33681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:44:00.416702 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40621
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:00.524937 16631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:00.525179 16631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:00.525219 16631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:00.529158 16631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:00.529211 16631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:00.529268 16631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:44:00.534552 16631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:00.535595 16631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:00.536558 16631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:00.545693 16639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:00.545693 16637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:00.545799 16631 server_base.cc:1034] running on GCE node
W20260928 02:44:00.545714 16636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:00.546329 16631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:00.546970 16631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:00.548132 16631 hybrid_clock.cc:648] HybridClock initialized: now 1790563440548103 us; error 39 us; skew 500 ppm
Sep 28 02:44:00 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:00.550740 16631 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:44:00.551571 16631 webserver.cc:460] Webserver started at http://127.2.123.3:43501/ using document root <none> and password file <none>
I20260928 02:44:00.552086 16631 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:00.552152 16631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:00.552351 16631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:00.554426 16631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "d74e56834e8740c1ac85a56a596ab2e1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "2d3b32091b8bbe0329d50cc253f81489"
server_key_iv: "275c8b20ea9a539998a423ae3cb8300d"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.554965 16631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "d74e56834e8740c1ac85a56a596ab2e1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "2d3b32091b8bbe0329d50cc253f81489"
server_key_iv: "275c8b20ea9a539998a423ae3cb8300d"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.558619 16631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:44:00.560885 16646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.562095 16631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:44:00.562194 16631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "d74e56834e8740c1ac85a56a596ab2e1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "2d3b32091b8bbe0329d50cc253f81489"
server_key_iv: "275c8b20ea9a539998a423ae3cb8300d"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.562279 16631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:00.586565 16631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:00.589262 16631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:00.589444 16631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:00.590059 16631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:00.590890 16631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:00.590938 16631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.590991 16631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:00.591018 16631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.599172 16631 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:46063
I20260928 02:44:00.599205 16759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:46063 every 8 connection(s)
I20260928 02:44:00.600159 16631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:00.602697 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 16631
I20260928 02:44:00.602829 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:44:00.603114 2540 external_mini_cluster.cc:1428] Setting key 0711182331a1942903ff26e879d23ea3
Sep 28 02:44:00 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:00.612901 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.601892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:45905 (local address 127.2.123.62:40621)
0928 02:44:00.602036 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.602040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.602871 (+ 831us) server_negotiation.cc:407] Connection header received
0928 02:44:00.604291 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.604295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.604351 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.604418 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:00.605558 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.606129 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.607238 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.607465 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.607551 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.609845 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:00.609849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:00.609851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:00.609876 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:00.611179 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:00.611637 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:00.611640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:00.611641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:00.611709 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:00.612026 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:00.612028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:00.612029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:00.612191 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:00.612259 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.612690 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.612782 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:44:00.613761 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.602282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:45905)
0928 02:44:00.602723 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.602767 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:00.604099 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:00.604598 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:00.604605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:00.604960 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:00.605422 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.605439 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.606246 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.606249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.607125 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.607135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.607596 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.607600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.607755 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.608243 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:00.608259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:00.609704 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:00.611282 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:00.611288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:00.611303 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:00.611559 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:00.611804 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:00.611806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:00.611808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:00.611947 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:00.612259 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:00.612264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:00.612504 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:44:00.612661 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:44:00.612913 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":127,"threads_started":1}
I20260928 02:44:00.614879 16760 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40621
I20260928 02:44:00.615120 16760 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:00.615666 16760 heartbeater.cc:511] Master 127.2.123.62:40621 requested a full tablet report, sending...
I20260928 02:44:00.616731 16301 ts_manager.cc:194] Registered new tserver with Master: d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063)
I20260928 02:44:00.617326 16301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:45905
I20260928 02:44:00.628345 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:44:00.636914 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.630168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.2.123.62:40621)
0928 02:44:00.630279 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.630284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.630444 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:44:00.630569 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.630572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.630617 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.630679 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:00.631359 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.631866 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.632682 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.632906 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.632996 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.634246 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:00.634250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:00.634253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:00.634279 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:00.635373 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:00.635755 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:00.635759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:00.635761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:00.635838 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:00.636081 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:00.636085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:00.636086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:00.636308 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:00.636429 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.636652 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.636760 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:44:00.639633 16301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:44:00.642657 16301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:00.656445 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.650596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:46063 (local address 127.0.0.1:45744)
0928 02:44:00.650954 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.651007 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:00.651132 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:00.652109 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:00.652116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:00.652138 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:00.652282 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.652289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.653549 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.653553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.654862 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.654871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.655899 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.655902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.656016 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.656060 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:44:00.656157 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:44:00.656280 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":46,"thread_start_us":83,"threads_started":1}
I20260928 02:44:00.656595 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.650696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.2.123.3:46063)
0928 02:44:00.651564 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.651576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.651600 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:44:00.651678 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.651683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.651915 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.652123 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:00.652383 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.653412 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.654997 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.655772 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.655886 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.656041 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.656369 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.656464 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":159,"threads_started":1}
I20260928 02:44:00.656819 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.651421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.2.123.2:41921)
0928 02:44:00.652190 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.652201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.652223 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:44:00.652294 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.652299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.652486 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.652625 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:00.653045 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.654072 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.655406 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.656036 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.656142 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.656280 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.656524 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.656629 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":107,"threads_started":1}
I20260928 02:44:00.657047 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.651755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.2.123.1:34991)
0928 02:44:00.652189 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.652202 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.652223 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:44:00.652294 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.652299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.652476 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.652626 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:00.652820 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.653678 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.655289 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.656134 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.656232 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.656374 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.656808 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.656905 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":93,"threads_started":1}
I20260928 02:44:00.657135 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.650932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41921 (local address 127.0.0.1:56412)
0928 02:44:00.651178 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.651190 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:00.651257 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:00.652802 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:00.652805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:00.652822 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:00.652919 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.652925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.654211 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.654215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.655305 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.655312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.656163 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.656167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.656261 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.656285 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:44:00.656346 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:44:00.656415 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
I20260928 02:44:00.657248 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.651340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:34991 (local address 127.0.0.1:49782)
0928 02:44:00.651862 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.651875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:00.651962 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:00.652610 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:00.652614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:00.652633 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:00.652714 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.652719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.653814 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.653818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.655140 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.655151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.656429 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.656433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.656509 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.656530 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:44:00.656590 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:44:00.656646 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260928 02:44:00.659788 16560 tablet_service.cc:1467] Processing CreateTablet for tablet a32e77e7f204462db1d657d7f1359bd5 (DEFAULT_TABLE table=test-table [id=9f9aa28f91b04820a3f8c97efced4877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:00.659844 16426 tablet_service.cc:1467] Processing CreateTablet for tablet a32e77e7f204462db1d657d7f1359bd5 (DEFAULT_TABLE table=test-table [id=9f9aa28f91b04820a3f8c97efced4877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:00.659827 16694 tablet_service.cc:1467] Processing CreateTablet for tablet a32e77e7f204462db1d657d7f1359bd5 (DEFAULT_TABLE table=test-table [id=9f9aa28f91b04820a3f8c97efced4877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:00.660742 16560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32e77e7f204462db1d657d7f1359bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:00.660826 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32e77e7f204462db1d657d7f1359bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:00.660820 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32e77e7f204462db1d657d7f1359bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:00.667716 16778 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Bootstrap starting.
I20260928 02:44:00.668150 16779 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: Bootstrap starting.
I20260928 02:44:00.668459 16777 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Bootstrap starting.
I20260928 02:44:00.670147 16778 tablet_bootstrap.cc:654] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:00.670732 16777 tablet_bootstrap.cc:654] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:00.670735 16779 tablet_bootstrap.cc:654] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:00.671173 16778 log.cc:824] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:00.671543 16777 log.cc:824] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:00.671562 16779 log.cc:824] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:00.673146 16778 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: No bootstrap required, opened a new log
I20260928 02:44:00.673345 16778 ts_tablet_manager.cc:1397] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:44:00.673468 16777 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: No bootstrap required, opened a new log
I20260928 02:44:00.673646 16777 ts_tablet_manager.cc:1397] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:44:00.673686 16779 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: No bootstrap required, opened a new log
I20260928 02:44:00.673856 16779 ts_tablet_manager.cc:1397] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:44:00.676710 16779 raft_consensus.cc:348] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.676955 16779 raft_consensus.cc:374] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:00.677035 16779 raft_consensus.cc:729] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74e56834e8740c1ac85a56a596ab2e1, State: Initialized, Role: FOLLOWER
I20260928 02:44:00.677217 16777 raft_consensus.cc:348] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.677217 16778 raft_consensus.cc:348] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.677477 16778 raft_consensus.cc:374] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:00.677477 16777 raft_consensus.cc:374] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:00.677536 16777 raft_consensus.cc:729] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c325a449b4cf4e8c8373333962c734c4, State: Initialized, Role: FOLLOWER
I20260928 02:44:00.677466 16779 consensus_queue.cc:260] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.677577 16778 raft_consensus.cc:729] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349efd8e11884710b85bbb42e632add1, State: Initialized, Role: FOLLOWER
I20260928 02:44:00.677974 16778 consensus_queue.cc:260] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.677987 16777 consensus_queue.cc:260] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.678146 16760 heartbeater.cc:503] Master 127.2.123.62:40621 was elected leader, sending a full tablet report...
I20260928 02:44:00.678483 16779 ts_tablet_manager.cc:1428] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:44:00.678911 16778 ts_tablet_manager.cc:1428] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:44:00.678911 16777 ts_tablet_manager.cc:1428] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:44:00.679224 16492 heartbeater.cc:503] Master 127.2.123.62:40621 was elected leader, sending a full tablet report...
I20260928 02:44:00.679224 16626 heartbeater.cc:503] Master 127.2.123.62:40621 was elected leader, sending a full tablet report...
I20260928 02:44:00.732775 16783 raft_consensus.cc:482] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:00.732982 16783 raft_consensus.cc:504] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.734162 16783 leader_election.cc:288] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991), c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921)
I20260928 02:44:00.738575 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.734477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:34095 (local address 127.2.123.1:34991)
0928 02:44:00.734621 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.734626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.734644 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:44:00.734723 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.734725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.734776 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.734853 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:00.735138 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.735872 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.737344 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.737953 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.738087 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.738151 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.738370 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.738446 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:44:00.738575 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.734527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:59143 (local address 127.2.123.2:41921)
0928 02:44:00.734659 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.734664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.735007 (+ 343us) server_negotiation.cc:407] Connection header received
0928 02:44:00.735106 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.735109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.735157 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.735232 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:00.735472 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.736109 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.737395 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.738006 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.738091 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.738150 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.738370 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.738467 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:44:00.738591 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.734368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:34991 (local address 127.2.123.3:34095)
0928 02:44:00.734509 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.734525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:00.734625 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:00.734877 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:00.734880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:00.734899 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:00.735032 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.735038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.736008 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.736010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.737217 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.737229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.738091 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.738095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.738230 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.738266 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:44:00.738338 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:44:00.738435 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":11}
I20260928 02:44:00.738602 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.734368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41921 (local address 127.2.123.3:59143)
0928 02:44:00.734919 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.734931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:00.735020 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:00.735244 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:00.735245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:00.735258 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:00.735347 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.735351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.736199 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.736202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.737299 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:00.737306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.738091 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.738095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:00.738226 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.738278 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:44:00.738338 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:44:00.738435 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":270,"thread_start_us":132,"threads_started":1}
I20260928 02:44:00.739452 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32e77e7f204462db1d657d7f1359bd5" candidate_uuid: "d74e56834e8740c1ac85a56a596ab2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c325a449b4cf4e8c8373333962c734c4" is_pre_election: true
I20260928 02:44:00.739472 16446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32e77e7f204462db1d657d7f1359bd5" candidate_uuid: "d74e56834e8740c1ac85a56a596ab2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349efd8e11884710b85bbb42e632add1" is_pre_election: true
I20260928 02:44:00.739779 16446 raft_consensus.cc:2454] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74e56834e8740c1ac85a56a596ab2e1 in term 0.
I20260928 02:44:00.739779 16580 raft_consensus.cc:2454] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74e56834e8740c1ac85a56a596ab2e1 in term 0.
I20260928 02:44:00.740257 16648 leader_election.cc:302] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c325a449b4cf4e8c8373333962c734c4, d74e56834e8740c1ac85a56a596ab2e1; no voters:
I20260928 02:44:00.740626 16783 raft_consensus.cc:2781] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:00.740705 16783 raft_consensus.cc:482] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:00.740733 16783 raft_consensus.cc:3037] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:00.741735 16783 raft_consensus.cc:504] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.742098 16783 leader_election.cc:288] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [CANDIDATE]: Term 1 election: Requested vote from peers 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991), c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921)
I20260928 02:44:00.742476 16446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32e77e7f204462db1d657d7f1359bd5" candidate_uuid: "d74e56834e8740c1ac85a56a596ab2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349efd8e11884710b85bbb42e632add1"
I20260928 02:44:00.742494 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32e77e7f204462db1d657d7f1359bd5" candidate_uuid: "d74e56834e8740c1ac85a56a596ab2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c325a449b4cf4e8c8373333962c734c4"
I20260928 02:44:00.742602 16580 raft_consensus.cc:3037] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:00.742594 16446 raft_consensus.cc:3037] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:00.743557 16580 raft_consensus.cc:2454] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74e56834e8740c1ac85a56a596ab2e1 in term 1.
I20260928 02:44:00.743564 16446 raft_consensus.cc:2454] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74e56834e8740c1ac85a56a596ab2e1 in term 1.
I20260928 02:44:00.743815 16649 leader_election.cc:302] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349efd8e11884710b85bbb42e632add1, d74e56834e8740c1ac85a56a596ab2e1; no voters:
I20260928 02:44:00.743963 16783 raft_consensus.cc:2781] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:00.744155 16783 raft_consensus.cc:686] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 LEADER]: Becoming Leader. State: Replica: d74e56834e8740c1ac85a56a596ab2e1, State: Running, Role: LEADER
I20260928 02:44:00.744478 16783 consensus_queue.cc:237] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:00.747620 16300 catalog_manager.cc:5495] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3). New cstate: current_term: 1 leader_uuid: "d74e56834e8740c1ac85a56a596ab2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } health_report { overall_health: UNKNOWN } } }
I20260928 02:44:00.778213 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.774757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.2.123.3:46063)
0928 02:44:00.774881 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:00.774884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:00.774898 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:44:00.775030 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:00.775032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:00.775085 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:00.775172 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:00.775386 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.775888 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.776736 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:00.776968 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:00.777051 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:00.777389 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:00.777828 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:00.777940 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:00.777984 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 02:44:00.778037 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:44:00.783944 16580 raft_consensus.cc:1261] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Refusing update from remote peer d74e56834e8740c1ac85a56a596ab2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:00.783962 16446 raft_consensus.cc:1261] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Refusing update from remote peer d74e56834e8740c1ac85a56a596ab2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:00.784617 16783 consensus_queue.cc:1035] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:00.784794 16787 consensus_queue.cc:1035] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:00.799010 16301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:00.799206 16301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:00.801739 16301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Sending DeleteTablet for 3 replicas of tablet a32e77e7f204462db1d657d7f1359bd5
I20260928 02:44:00.802544 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 16363
I20260928 02:44:00.802539 16560 tablet_service.cc:1514] Processing DeleteTablet for tablet a32e77e7f204462db1d657d7f1359bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:00 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56412
I20260928 02:44:00.802585 16694 tablet_service.cc:1514] Processing DeleteTablet for tablet a32e77e7f204462db1d657d7f1359bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:00 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:45744
I20260928 02:44:00.803699 16801 tablet_replica.cc:330] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: stopping tablet replica
I20260928 02:44:00.803946 16800 tablet_replica.cc:330] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: stopping tablet replica
I20260928 02:44:00.804280 16800 raft_consensus.cc:2229] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:00.804570 16800 raft_consensus.cc:2258] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:00.804953 16801 raft_consensus.cc:2229] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:00.805300 16801 raft_consensus.cc:2258] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:00.806572 16801 ts_tablet_manager.cc:1905] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:00.809691 16800 ts_tablet_manager.cc:1905] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:44:00.809705 16287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:34991 (error 108)
I20260928 02:44:00.809947 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 16497
I20260928 02:44:00.810936 16801 ts_tablet_manager.cc:1918] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:00.811058 16801 log.cc:1196] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/a32e77e7f204462db1d657d7f1359bd5
I20260928 02:44:00.811074 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.810732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:34991 (local address 127.0.0.1:49792)
0928 02:44:00.810908 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.810982 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260928 02:44:00.811358 16287 catalog_manager.cc:4584] TS 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a32e77e7f204462db1d657d7f1359bd5: Network error: Client connection negotiation failed: client connection to 127.2.123.1:34991: connect: Connection refused (error 111)
I20260928 02:44:00.811411 16801 ts_tablet_manager.cc:1939] T a32e77e7f204462db1d657d7f1359bd5 P d74e56834e8740c1ac85a56a596ab2e1: Deleting consensus metadata
I20260928 02:44:00.812422 16287 catalog_manager.cc:4841] TS d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063): tablet a32e77e7f204462db1d657d7f1359bd5 (table test-table [id=9f9aa28f91b04820a3f8c97efced4877]) successfully deleted
I20260928 02:44:00.817699 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 16631
I20260928 02:44:00.819352 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.817831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41921 (local address 127.0.0.1:56414)
0928 02:44:00.819233 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.819281 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.2:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1325}
I20260928 02:44:00.822397 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 16269
I20260928 02:44:00.827523 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40621
--webserver_interface=127.2.123.62
--webserver_port=38225
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:00.935621 16803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:00.935883 16803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:00.935922 16803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:00.939913 16803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:00.939986 16803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:00.940006 16803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:00.940021 16803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:00.940034 16803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:00.945304 16803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:00.946385 16803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:00.947417 16803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:00.959914 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:00.959924 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:00.960094 16803 server_base.cc:1034] running on GCE node
W20260928 02:44:00.960251 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:00.960872 16803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:00.962028 16803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:00.963203 16803 hybrid_clock.cc:648] HybridClock initialized: now 1790563440963177 us; error 40 us; skew 500 ppm
Sep 28 02:44:00 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:00.965893 16803 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:44:00.966851 16803 webserver.cc:460] Webserver started at http://127.2.123.62:38225/ using document root <none> and password file <none>
I20260928 02:44:00.967383 16803 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:00.967458 16803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:00.971292 16803 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 02:44:00.973526 16819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:00.974807 16803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:44:00.974913 16803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "692fc8a19f074f9c83a0197417fe5a96"
format_stamp: "Formatted at 2026-09-28 02:43:59 on dist-test-slave-rqvb"
server_key: "8b2b8d922f1e24ea8acdcb274fb64909"
server_key_iv: "6172a777a47a3e70bd42229c47cc104a"
server_key_version: "encryptionkey@0"
I20260928 02:44:00.975548 16803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:00.988678 16803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:00.991881 16803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:00.992058 16803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:00.999208 16803 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:40621
I20260928 02:44:00.999481 16871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:40621 every 8 connection(s)
I20260928 02:44:01.000308 16803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:01.004101 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 16803
I20260928 02:44:01.004570 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:34991
--local_ip_for_outbound_sockets=127.2.123.1
--tserver_master_addrs=127.2.123.62:40621
--webserver_port=41447
--webserver_interface=127.2.123.1
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:01.004940 16872 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:44:01.006868 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Bootstrap starting.
I20260928 02:44:01.023366 16872 log.cc:824] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:01.032135 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:01.032490 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Bootstrap complete.
I20260928 02:44:01.036309 16872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:44:01.036701 16872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692fc8a19f074f9c83a0197417fe5a96, State: Initialized, Role: FOLLOWER
I20260928 02:44:01.037145 16872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:44:01.037254 16872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:01.037319 16872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:01.037418 16872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:44:01.039306 16872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:44:01.039623 16872 leader_election.cc:302] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692fc8a19f074f9c83a0197417fe5a96; no voters:
I20260928 02:44:01.039923 16872 leader_election.cc:288] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:44:01.040030 16876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:44:01.040225 16876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 2 LEADER]: Becoming Leader. State: Replica: 692fc8a19f074f9c83a0197417fe5a96, State: Running, Role: LEADER
I20260928 02:44:01.040511 16876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:44:01.041157 16872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:01.041455 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692fc8a19f074f9c83a0197417fe5a96. Latest consensus state: current_term: 2 leader_uuid: "692fc8a19f074f9c83a0197417fe5a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } } }
I20260928 02:44:01.041653 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:01.042292 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "692fc8a19f074f9c83a0197417fe5a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } } }
I20260928 02:44:01.042451 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:01.042910 16880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:01.047508 16880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:01.048142 16880 catalog_manager.cc:1259] Loaded cluster ID: 688465d0ea3649919a5ab57e6753a91e
I20260928 02:44:01.048202 16880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:44:01.049669 16880 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:44:01.054435 16880 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:44:01.059922 16880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:01.060711 16880 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Loaded TSK: 0
I20260928 02:44:01.061950 16880 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:44:01.116982 16874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:01.117260 16874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:01.117321 16874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:01.121407 16874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:01.121465 16874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:01.121543 16874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:44:01.126796 16874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:34991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=41447
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:01.127780 16874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:01.128816 16874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:01.138463 16900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:01.138620 16874 server_base.cc:1034] running on GCE node
W20260928 02:44:01.138463 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:01.138886 16902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:01.139307 16874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:01.139942 16874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:01.141116 16874 hybrid_clock.cc:648] HybridClock initialized: now 1790563441141090 us; error 42 us; skew 500 ppm
Sep 28 02:44:01 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:01.144052 16874 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:44:01.144989 16874 webserver.cc:460] Webserver started at http://127.2.123.1:41447/ using document root <none> and password file <none>
I20260928 02:44:01.145533 16874 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:01.145617 16874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:01.149673 16874 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:44:01.152007 16909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:01.153378 16874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:44:01.153486 16874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "349efd8e11884710b85bbb42e632add1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "d87e645cd2d04af52731449bf31f819e"
server_key_iv: "5e59f9a89d4ec6dfc3b4d8df558d5e0c"
server_key_version: "encryptionkey@0"
I20260928 02:44:01.154058 16874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:01.182734 16874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:01.185940 16874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:01.186146 16874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:01.186772 16874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:01.187814 16916 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:44:01.191933 16874 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:44:01.192013 16874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 02:44:01.192090 16874 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:44:01.193876 16874 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:44:01.193925 16874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 02:44:01.194332 16916 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Bootstrap starting.
I20260928 02:44:01.203056 16874 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:34991
I20260928 02:44:01.203181 17023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:34991 every 8 connection(s)
I20260928 02:44:01.204042 16874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:01.211388 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 16874
I20260928 02:44:01.211848 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:41921
--local_ip_for_outbound_sockets=127.2.123.2
--tserver_master_addrs=127.2.123.62:40621
--webserver_port=41371
--webserver_interface=127.2.123.2
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:01.213500 16916 log.cc:824] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:44:01 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:01.221313 16916 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:01.221722 16916 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Bootstrap complete.
I20260928 02:44:01.222976 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.205910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:40317)
0928 02:44:01.206517 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.206574 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:01.207684 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:01.208864 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:01.208873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:01.209375 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:01.210164 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.210186 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.211684 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.211688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.212946 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.212958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.213995 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.213999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.214202 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.214801 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:01.214821 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:01.216693 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:01.219343 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.219349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.219362 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.219689 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.220135 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.220138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.220140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.220312 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.221058 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:01.221064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:01.221365 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 02:44:01.221556 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 02:44:01.221886 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":162,"threads_started":1}
I20260928 02:44:01.223026 16916 ts_tablet_manager.cc:1397] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.016s
I20260928 02:44:01.223450 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.206098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:40317 (local address 127.2.123.62:40621)
0928 02:44:01.206540 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:01.206558 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:01.206699 (+ 141us) server_negotiation.cc:407] Connection header received
0928 02:44:01.208126 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:01.208137 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:01.208446 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:01.208712 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:01.210357 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.211497 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.213102 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.213825 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.214029 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.216931 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:01.216938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:01.216958 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:01.216996 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:01.219201 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.219832 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.219840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.219847 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.220033 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.220425 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.220429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.220431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.220930 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:01.221075 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:01.221730 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 02:44:01.222152 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":100,"threads_started":1}
I20260928 02:44:01.225987 17024 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40621
I20260928 02:44:01.225595 16916 raft_consensus.cc:348] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:01.226140 16916 raft_consensus.cc:729] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349efd8e11884710b85bbb42e632add1, State: Initialized, Role: FOLLOWER
I20260928 02:44:01.226233 17024 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:01.226608 16916 consensus_queue.cc:260] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:01.226796 17024 heartbeater.cc:511] Master 127.2.123.62:40621 requested a full tablet report, sending...
I20260928 02:44:01.227478 16916 ts_tablet_manager.cc:1428] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 02:44:01.229387 16836 ts_manager.cc:194] Registered new tserver with Master: 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991)
I20260928 02:44:01.232867 16836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:40317
I20260928 02:44:01.235531 17024 heartbeater.cc:503] Master 127.2.123.62:40621 was elected leader, sending a full tablet report...
Sep 28 02:44:01 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:44:01.244521 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.232365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.2.123.1:34991)
0928 02:44:01.233584 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:01.233595 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:01.233616 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:44:01.233690 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:01.233695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:01.233916 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:01.234085 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:01.234868 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.235858 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.237619 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.237997 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.238093 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.240487 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:01.240499 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:01.240509 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:01.240533 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:01.242440 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.242977 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.242982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.242983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.243076 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.243363 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.243367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.243368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.243734 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:01.243865 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:01.244232 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:44:01.244389 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":121,"threads_started":1}
I20260928 02:44:01.244524 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.232252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:34991 (local address 127.0.0.1:49800)
0928 02:44:01.233139 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.233185 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:01.233313 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:01.234092 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:01.234098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:01.234562 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:01.234712 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.234721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.235964 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.235967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.237511 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.237522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.238097 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.238100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.238208 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.238708 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:01.238720 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:01.240323 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:01.242543 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.242553 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.242558 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.242867 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.243168 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.243171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.243173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.243267 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.243859 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:01.243865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:01.243996 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 02:44:01.244137 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:44:01.244335 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":130,"threads_started":1}
I20260928 02:44:01.245672 16958 tablet_service.cc:1514] Processing DeleteTablet for tablet a32e77e7f204462db1d657d7f1359bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:00 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:49800
I20260928 02:44:01.246613 17033 tablet_replica.cc:330] stopping tablet replica
I20260928 02:44:01.246935 17033 raft_consensus.cc:2229] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:01.247211 17033 raft_consensus.cc:2258] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:01.248320 17033 ts_tablet_manager.cc:1905] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:01.252857 17033 ts_tablet_manager.cc:1918] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:01.252969 17033 log.cc:1196] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/a32e77e7f204462db1d657d7f1359bd5
I20260928 02:44:01.253242 17033 ts_tablet_manager.cc:1939] T a32e77e7f204462db1d657d7f1359bd5 P 349efd8e11884710b85bbb42e632add1: Deleting consensus metadata
I20260928 02:44:01.255333 16822 catalog_manager.cc:4841] TS 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991): tablet a32e77e7f204462db1d657d7f1359bd5 (table test-table [id=9f9aa28f91b04820a3f8c97efced4877]) successfully deleted
W20260928 02:44:01.331403 17028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:01.331698 17028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:01.331743 17028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:01.336169 17028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:01.336254 17028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:01.336323 17028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:44:01.342020 17028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:41921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=41371
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:01.343133 17028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:01.344118 17028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:01.353631 17041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:01.353703 17028 server_base.cc:1034] running on GCE node
W20260928 02:44:01.353649 17039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:01.353636 17038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:01.354348 17028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:01.354967 17028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:01.356139 17028 hybrid_clock.cc:648] HybridClock initialized: now 1790563441356121 us; error 32 us; skew 500 ppm
Sep 28 02:44:01 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:01.358935 17028 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:44:01.359836 17028 webserver.cc:460] Webserver started at http://127.2.123.2:41371/ using document root <none> and password file <none>
I20260928 02:44:01.360374 17028 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:01.360455 17028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:01.364387 17028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:44:01.366740 17048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:01.368016 17028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:44:01.368124 17028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "c325a449b4cf4e8c8373333962c734c4"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "b71aa98ef4115a4722ae5244e426a8f0"
server_key_iv: "70d348c4e578c39e9f5863812af8ff72"
server_key_version: "encryptionkey@0"
I20260928 02:44:01.368674 17028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:01.379590 17028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:01.387498 17028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:01.387676 17028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:01.388252 17028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:01.389374 17055 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:44:01.393427 17028 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:44:01.393497 17028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 02:44:01.393563 17028 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:44:01.395264 17028 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:44:01.395311 17028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 02:44:01.395653 17055 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Bootstrap starting.
I20260928 02:44:01.401243 17055 log.cc:824] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:01.404528 17028 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:41921
I20260928 02:44:01.404637 17163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:41921 every 8 connection(s)
I20260928 02:44:01.405352 17028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:01.409176 17055 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:01.409180 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 17028
I20260928 02:44:01.409535 17055 tablet_bootstrap.cc:492] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Bootstrap complete.
I20260928 02:44:01.409610 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:46063
--local_ip_for_outbound_sockets=127.2.123.3
--tserver_master_addrs=127.2.123.62:40621
--webserver_port=43501
--webserver_interface=127.2.123.3
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:01.409883 17055 ts_tablet_manager.cc:1397] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20260928 02:44:01.412446 17055 raft_consensus.cc:348] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:01.412870 17055 raft_consensus.cc:729] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c325a449b4cf4e8c8373333962c734c4, State: Initialized, Role: FOLLOWER
I20260928 02:44:01.413367 17055 consensus_queue.cc:260] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } }
I20260928 02:44:01.414388 17055 ts_tablet_manager.cc:1428] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Time spent starting tablet: real 0.004s user 0.008s sys 0.000s
Sep 28 02:44:01 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:01.419754 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.407091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:58179 (local address 127.2.123.62:40621)
0928 02:44:01.407248 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:01.407252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:01.407836 (+ 584us) server_negotiation.cc:407] Connection header received
0928 02:44:01.408774 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:01.408778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:01.408835 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:01.408939 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:01.410387 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.411188 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.412753 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.413096 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.413204 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.416277 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:01.416281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:01.416284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:01.416317 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:01.417693 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.418254 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.418258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.418260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.418342 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.418666 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.418669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.418671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.418860 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:01.418946 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:01.419451 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 02:44:01.419585 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:44:01.420599 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.407197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:58179)
0928 02:44:01.407672 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.407723 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:01.408566 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:01.409087 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:01.409095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:01.409554 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:01.410221 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.410241 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.411368 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.411372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.412619 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.412631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.413233 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.413237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.413426 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.414495 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:01.414516 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:01.416105 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:01.417810 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.417816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.417831 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.418151 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.418427 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.418431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.418433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.418572 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.418958 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:01.418964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:01.419210 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 02:44:01.419368 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 02:44:01.419699 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
I20260928 02:44:01.421810 17164 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40621
I20260928 02:44:01.422072 17164 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:01.422582 17164 heartbeater.cc:511] Master 127.2.123.62:40621 requested a full tablet report, sending...
I20260928 02:44:01.424275 16836 ts_manager.cc:194] Registered new tserver with Master: c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921)
I20260928 02:44:01.425539 16836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:58179
I20260928 02:44:01.427744 17164 heartbeater.cc:503] Master 127.2.123.62:40621 was elected leader, sending a full tablet report...
Sep 28 02:44:01 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563440, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.2@KRBTEST.COM
I20260928 02:44:01.434666 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.425183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41921 (local address 127.0.0.1:56416)
0928 02:44:01.425371 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.425387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:01.425468 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:01.426178 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:01.426181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:01.426375 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:01.426482 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.426488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.427787 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.427791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.428677 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.428694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.429438 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.429442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.429541 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.430008 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:01.430021 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:01.431338 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:01.433073 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.433077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.433079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.433314 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.433569 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.433572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.433574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.433661 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.434162 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:01.434165 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:01.434269 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:44:01.434366 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:44:01.434506 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 02:44:01.434749 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.425280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.2.123.2:41921)
0928 02:44:01.425699 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:01.425709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:01.425726 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:44:01.425786 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:01.425792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:01.426063 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:01.426195 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:01.426599 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.427663 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.428801 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.429330 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.429420 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.431468 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:01.431476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:01.431488 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:01.431509 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:01.432952 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.433416 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.433421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.433422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.433499 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.433779 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.433783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.433784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.434074 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:01.434180 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:01.434474 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 02:44:01.434608 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260928 02:44:01.435711 17097 tablet_service.cc:1514] Processing DeleteTablet for tablet a32e77e7f204462db1d657d7f1359bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:00 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56416
I20260928 02:44:01.436726 17170 tablet_replica.cc:330] stopping tablet replica
I20260928 02:44:01.437001 17170 raft_consensus.cc:2229] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:01.437227 17170 raft_consensus.cc:2258] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:01.438210 17170 ts_tablet_manager.cc:1905] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:01.441828 17170 ts_tablet_manager.cc:1918] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:01.441916 17170 log.cc:1196] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/a32e77e7f204462db1d657d7f1359bd5
I20260928 02:44:01.442183 17170 ts_tablet_manager.cc:1939] T a32e77e7f204462db1d657d7f1359bd5 P c325a449b4cf4e8c8373333962c734c4: Deleting consensus metadata
I20260928 02:44:01.443784 16821 catalog_manager.cc:4841] TS c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921): tablet a32e77e7f204462db1d657d7f1359bd5 (table test-table [id=9f9aa28f91b04820a3f8c97efced4877]) successfully deleted
W20260928 02:44:01.523185 17167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:01.523504 17167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:01.523576 17167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:01.528424 17167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:01.528492 17167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:01.528576 17167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:44:01.533929 17167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:46063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=43501
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:01.535020 17167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:01.536043 17167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:01.545681 17176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:01.545671 17175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:01.545677 17167 server_base.cc:1034] running on GCE node
W20260928 02:44:01.545710 17178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:01.546321 17167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:01.546967 17167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:01.548158 17167 hybrid_clock.cc:648] HybridClock initialized: now 1790563441548134 us; error 37 us; skew 500 ppm
Sep 28 02:44:01 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:01.550957 17167 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:44:01.551805 17167 webserver.cc:460] Webserver started at http://127.2.123.3:43501/ using document root <none> and password file <none>
I20260928 02:44:01.552322 17167 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:01.552400 17167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:01.556304 17167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:44:01.558597 17185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:01.559955 17167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 02:44:01.560065 17167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "d74e56834e8740c1ac85a56a596ab2e1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "2d3b32091b8bbe0329d50cc253f81489"
server_key_iv: "275c8b20ea9a539998a423ae3cb8300d"
server_key_version: "encryptionkey@0"
I20260928 02:44:01.560602 17167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:01.574433 17167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:01.577833 17167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:01.578043 17167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:01.578686 17167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:01.579576 17167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:01.579630 17167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:01.579670 17167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:01.579690 17167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:01.588116 17167 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:46063
I20260928 02:44:01.588142 17298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:46063 every 8 connection(s)
I20260928 02:44:01.589121 17167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:01.596398 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 17167
Sep 28 02:44:01 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:01.602456 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.590921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:33009 (local address 127.2.123.62:40621)
0928 02:44:01.591075 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:01.591079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:01.591930 (+ 851us) server_negotiation.cc:407] Connection header received
0928 02:44:01.593222 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:01.593225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:01.593284 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:01.593362 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:01.594610 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.595141 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.596252 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.596493 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.596587 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.599099 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:01.599102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:01.599104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:01.599128 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:01.600422 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.600963 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.600965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.600967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.601043 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.601401 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.601404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.601405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.601582 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:01.601676 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:01.602200 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 02:44:01.602314 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60,"spinlock_wait_cycles":1792}
I20260928 02:44:01.603358 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.591331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:33009)
0928 02:44:01.591771 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.591817 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:01.593011 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:01.593570 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:01.593576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:01.593943 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:01.594464 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.594478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.595261 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.595264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.596134 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.596143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.596634 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.596638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.596800 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.597422 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:01.597438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:01.598928 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:01.600525 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.600531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.600546 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.600858 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.601121 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:01.601124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:01.601126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:01.601306 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:01.601686 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:01.601692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:01.601931 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 02:44:01.602125 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 02:44:01.602407 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
I20260928 02:44:01.604650 17299 heartbeater.cc:347] Connected to a master server at 127.2.123.62:40621
I20260928 02:44:01.604899 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.597054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.2.123.62:40621)
0928 02:44:01.597323 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:01.597327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:01.597443 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:44:01.597584 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:01.597587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:01.597628 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:01.597698 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:01.598444 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.599166 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.599994 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.600223 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.600306 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.601599 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:01.601603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:01.601605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:01.601625 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:01.603219 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.603692 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.603696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.603697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.603773 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:01.604044 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:01.604048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:01.604050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:01.604248 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:01.604440 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:01.604609 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:44:01.604737 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"spinlock_wait_cycles":6400,"thread_start_us":64,"threads_started":1}
I20260928 02:44:01.604923 17299 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:01.605486 17299 heartbeater.cc:511] Master 127.2.123.62:40621 requested a full tablet report, sending...
I20260928 02:44:01.606564 16835 ts_manager.cc:194] Registered new tserver with Master: d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063)
I20260928 02:44:01.607136 16835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:33009
I20260928 02:44:01.619596 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:44:01.624099 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.620261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.2.123.62:40621)
0928 02:44:01.620400 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:01.620406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:01.620694 (+ 288us) server_negotiation.cc:407] Connection header received
0928 02:44:01.620811 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:01.620813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:01.620930 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:01.621023 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:01.621227 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.621745 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.622605 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.622840 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.622932 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.623306 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:01.623710 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:01.623796 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:01.623898 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:44:01.623953 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:44:01.624815 16835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:44:01.628124 16835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:01.635615 17097 tablet_service.cc:1467] Processing CreateTablet for tablet 62040feea41a4d7992a6843b30f6b911 (DEFAULT_TABLE table=test-table [id=df8b6e94848540c38a9999e648a1c103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:01.635705 16958 tablet_service.cc:1467] Processing CreateTablet for tablet 62040feea41a4d7992a6843b30f6b911 (DEFAULT_TABLE table=test-table [id=df8b6e94848540c38a9999e648a1c103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:01.636269 17097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62040feea41a4d7992a6843b30f6b911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:01.636395 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62040feea41a4d7992a6843b30f6b911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:01.639112 17055 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Bootstrap starting.
I20260928 02:44:01.639457 16916 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Bootstrap starting.
I20260928 02:44:01.640071 17055 tablet_bootstrap.cc:654] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:01.640333 16916 tablet_bootstrap.cc:654] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:01.640537 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.634748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:46063 (local address 127.0.0.1:45772)
0928 02:44:01.634950 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.634965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:01.635043 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:01.636067 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:01.636070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:01.636089 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:01.636197 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.636203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.637523 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.637527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.638787 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:01.638801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.640037 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.640041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:01.640138 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.640196 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:44:01.640287 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:44:01.640380 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260928 02:44:01.640589 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.635067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.2.123.3:46063)
0928 02:44:01.635478 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:01.635491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:01.635516 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:44:01.635593 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:01.635599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:01.635848 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:01.636060 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:01.636304 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.637373 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.638933 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:01.639902 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:01.640019 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:01.640172 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:01.640333 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:44:01.640438 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":180,"threads_started":1}
I20260928 02:44:01.641870 17055 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: No bootstrap required, opened a new log
I20260928 02:44:01.641952 17055 ts_tablet_manager.cc:1397] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:44:01.642445 17055 raft_consensus.cc:348] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:01.642560 17055 raft_consensus.cc:374] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:01.642589 17055 raft_consensus.cc:729] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c325a449b4cf4e8c8373333962c734c4, State: Initialized, Role: FOLLOWER
I20260928 02:44:01.642714 17055 consensus_queue.cc:260] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:01.642966 17055 ts_tablet_manager.cc:1428] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:44:01.643764 16916 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: No bootstrap required, opened a new log
I20260928 02:44:01.643857 16916 ts_tablet_manager.cc:1397] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260928 02:44:01.644259 16916 raft_consensus.cc:348] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:01.644382 16916 raft_consensus.cc:374] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:01.644423 16916 raft_consensus.cc:729] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349efd8e11884710b85bbb42e632add1, State: Initialized, Role: FOLLOWER
I20260928 02:44:01.644542 16916 consensus_queue.cc:260] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:01.644076 17233 tablet_service.cc:1467] Processing CreateTablet for tablet 62040feea41a4d7992a6843b30f6b911 (DEFAULT_TABLE table=test-table [id=df8b6e94848540c38a9999e648a1c103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:01.644770 16916 ts_tablet_manager.cc:1428] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:44:01.645277 17233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62040feea41a4d7992a6843b30f6b911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:01.653648 17311 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Bootstrap starting.
I20260928 02:44:01.656095 17311 tablet_bootstrap.cc:654] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Neither blocks nor log segments found. Creating new log.
W20260928 02:44:01.656370 17165 tablet.cc:2369] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:44:01.657011 17311 log.cc:824] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:01.659024 17311 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: No bootstrap required, opened a new log
I20260928 02:44:01.659215 17311 ts_tablet_manager.cc:1397] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:44:01.662314 17311 raft_consensus.cc:348] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:01.662607 17311 raft_consensus.cc:374] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:01.662670 17311 raft_consensus.cc:729] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74e56834e8740c1ac85a56a596ab2e1, State: Initialized, Role: FOLLOWER
I20260928 02:44:01.663100 17311 consensus_queue.cc:260] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:01.663724 17299 heartbeater.cc:503] Master 127.2.123.62:40621 was elected leader, sending a full tablet report...
I20260928 02:44:01.664059 17311 ts_tablet_manager.cc:1428] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 02:44:01.705269 17025 tablet.cc:2369] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:44:01.840055 17300 tablet.cc:2369] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:44:02.019683 17313 raft_consensus.cc:482] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:02.019924 17313 raft_consensus.cc:504] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:02.021103 17313 leader_election.cc:288] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921), 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991)
I20260928 02:44:02.026743 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.021435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41921 (local address 127.2.123.3:48073)
0928 02:44:02.021656 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.021671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:02.021770 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:02.022148 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:02.022150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:02.022163 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:02.022279 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:02.022285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.023422 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.023426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:02.024932 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:02.024946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.026041 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.026045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:02.026234 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:02.026347 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 02:44:02.026439 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:44:02.026545 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":216}
I20260928 02:44:02.026759 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.021590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:36005 (local address 127.2.123.1:34991)
0928 02:44:02.021902 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:02.021907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:02.022027 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:44:02.022131 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:02.022134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:02.022202 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:02.022306 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:02.022538 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.023299 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.025114 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.025883 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.026019 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:02.026212 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:02.026505 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 02:44:02.026600 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":95,"threads_started":1}
I20260928 02:44:02.026788 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.021521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:48073 (local address 127.2.123.2:41921)
0928 02:44:02.021811 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:02.021816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:02.021836 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:44:02.021898 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:02.021901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:02.022006 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:02.022138 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:02.022402 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.023266 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.025114 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.025862 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.026020 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:02.026204 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:02.026500 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:44:02.026600 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260928 02:44:02.026798 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.021476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:34991 (local address 127.2.123.3:36005)
0928 02:44:02.021922 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.021932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:02.022033 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:02.022324 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:02.022326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:02.022343 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:02.022436 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:02.022440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.023422 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.023426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:02.024932 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:02.024946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.026041 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.026045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:02.026349 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:02.026367 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:44:02.026439 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:44:02.026545 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":210,"thread_start_us":120,"threads_started":1}
I20260928 02:44:02.027489 17117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62040feea41a4d7992a6843b30f6b911" candidate_uuid: "d74e56834e8740c1ac85a56a596ab2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c325a449b4cf4e8c8373333962c734c4" is_pre_election: true
I20260928 02:44:02.027848 17117 raft_consensus.cc:2454] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74e56834e8740c1ac85a56a596ab2e1 in term 0.
I20260928 02:44:02.027808 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62040feea41a4d7992a6843b30f6b911" candidate_uuid: "d74e56834e8740c1ac85a56a596ab2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349efd8e11884710b85bbb42e632add1" is_pre_election: true
I20260928 02:44:02.028079 16978 raft_consensus.cc:2454] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74e56834e8740c1ac85a56a596ab2e1 in term 0.
I20260928 02:44:02.028352 17187 leader_election.cc:302] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c325a449b4cf4e8c8373333962c734c4, d74e56834e8740c1ac85a56a596ab2e1; no voters:
I20260928 02:44:02.028676 17313 raft_consensus.cc:2781] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:02.028764 17313 raft_consensus.cc:482] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:02.028815 17313 raft_consensus.cc:3037] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:02.029989 17313 raft_consensus.cc:504] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:02.030347 17313 leader_election.cc:288] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [CANDIDATE]: Term 1 election: Requested vote from peers c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921), 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991)
I20260928 02:44:02.030774 17117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62040feea41a4d7992a6843b30f6b911" candidate_uuid: "d74e56834e8740c1ac85a56a596ab2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c325a449b4cf4e8c8373333962c734c4"
I20260928 02:44:02.030891 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62040feea41a4d7992a6843b30f6b911" candidate_uuid: "d74e56834e8740c1ac85a56a596ab2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349efd8e11884710b85bbb42e632add1"
I20260928 02:44:02.030912 17117 raft_consensus.cc:3037] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:02.031028 16978 raft_consensus.cc:3037] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:02.032014 17117 raft_consensus.cc:2454] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74e56834e8740c1ac85a56a596ab2e1 in term 1.
I20260928 02:44:02.032016 16978 raft_consensus.cc:2454] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74e56834e8740c1ac85a56a596ab2e1 in term 1.
I20260928 02:44:02.032364 17188 leader_election.cc:302] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 349efd8e11884710b85bbb42e632add1, c325a449b4cf4e8c8373333962c734c4, d74e56834e8740c1ac85a56a596ab2e1; no voters:
I20260928 02:44:02.032536 17313 raft_consensus.cc:2781] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:02.032747 17313 raft_consensus.cc:686] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 LEADER]: Becoming Leader. State: Replica: d74e56834e8740c1ac85a56a596ab2e1, State: Running, Role: LEADER
I20260928 02:44:02.033078 17313 consensus_queue.cc:237] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:02.036614 16836 catalog_manager.cc:5495] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3). New cstate: current_term: 1 leader_uuid: "d74e56834e8740c1ac85a56a596ab2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } health_report { overall_health: UNKNOWN } } }
I20260928 02:44:02.099325 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.095844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.2.123.3:46063)
0928 02:44:02.095986 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:02.095991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:02.096005 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:44:02.096120 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:02.096122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:02.096183 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:02.096322 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:02.096583 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.097092 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.097912 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:02.098146 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:02.098238 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:02.098515 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:02.098929 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:02.099037 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:02.099103 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:44:02.099164 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:44:02.105338 16978 raft_consensus.cc:1261] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Refusing update from remote peer d74e56834e8740c1ac85a56a596ab2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:02.105331 17117 raft_consensus.cc:1261] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Refusing update from remote peer d74e56834e8740c1ac85a56a596ab2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:02.105949 17317 consensus_queue.cc:1035] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:02.106194 17313 consensus_queue.cc:1035] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:02.110826 17319 mvcc.cc:205] Tried to move back new op lower bound from 7334147858856026112 to 7334147858570432512. Current Snapshot: MvccSnapshot[applied={T|T < 7334147858856026112}]
I20260928 02:44:02.112313 17321 mvcc.cc:205] Tried to move back new op lower bound from 7334147858856026112 to 7334147858570432512. Current Snapshot: MvccSnapshot[applied={T|T < 7334147858856026112}]
I20260928 02:44:02.120055 16835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:02.120272 16835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:02.123032 16835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Sending DeleteTablet for 3 replicas of tablet 62040feea41a4d7992a6843b30f6b911
I20260928 02:44:02.123805 17097 tablet_service.cc:1514] Processing DeleteTablet for tablet 62040feea41a4d7992a6843b30f6b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:02 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56416
I20260928 02:44:02.123824 16958 tablet_service.cc:1514] Processing DeleteTablet for tablet 62040feea41a4d7992a6843b30f6b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:02 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:49800
I20260928 02:44:02.123910 17233 tablet_service.cc:1514] Processing DeleteTablet for tablet 62040feea41a4d7992a6843b30f6b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:02 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:45772
I20260928 02:44:02.124241 17328 tablet_replica.cc:330] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: stopping tablet replica
I20260928 02:44:02.124367 17328 raft_consensus.cc:2229] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:02.124444 17330 tablet_replica.cc:330] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: stopping tablet replica
I20260928 02:44:02.124502 17328 raft_consensus.cc:2258] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:02.124806 17330 raft_consensus.cc:2229] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:02.125000 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 16874
I20260928 02:44:02.126533 17330 raft_consensus.cc:2258] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:02.128306 17330 ts_tablet_manager.cc:1905] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:44:02.130124 16822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:34991 (error 108)
I20260928 02:44:02.131186 17330 ts_tablet_manager.cc:1918] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:02.131305 17330 log.cc:1196] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/62040feea41a4d7992a6843b30f6b911
I20260928 02:44:02.131665 17330 ts_tablet_manager.cc:1939] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Deleting consensus metadata
W20260928 02:44:02.132100 17031 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:02.131091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:34991 (local address 127.0.0.1:49812)
0928 02:44:02.131220 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.131237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:02.131377 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:02.131973 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:34991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:44:02.132136 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 17028
W20260928 02:44:02.133064 16822 catalog_manager.cc:4584] TS 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62040feea41a4d7992a6843b30f6b911: Network error: Client connection negotiation failed: client connection to 127.2.123.1:34991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:44:02.134409 16822 catalog_manager.cc:4841] TS d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063): tablet 62040feea41a4d7992a6843b30f6b911 (table test-table [id=df8b6e94848540c38a9999e648a1c103]) successfully deleted
W20260928 02:44:02.137876 17031 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:02.137536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41921 (local address 127.0.0.1:56428)
0928 02:44:02.137712 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.137787 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.2:41921: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 02:44:02.137936 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 17167
I20260928 02:44:02.143090 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 16803
I20260928 02:44:02.148238 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40621
--webserver_interface=127.2.123.62
--webserver_port=38225
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:02.260644 17332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:02.260947 17332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:02.261005 17332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:02.265092 17332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:02.265167 17332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:02.265194 17332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:02.265221 17332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:02.265244 17332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:02.270778 17332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:02.271915 17332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:02.272981 17332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:02.281764 17337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:02.281883 17332 server_base.cc:1034] running on GCE node
W20260928 02:44:02.281800 17340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:02.281782 17338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:02.282701 17332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:02.283923 17332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:02.285107 17332 hybrid_clock.cc:648] HybridClock initialized: now 1790563442285082 us; error 37 us; skew 500 ppm
Sep 28 02:44:02 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:02.288043 17332 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:44:02.288962 17332 webserver.cc:460] Webserver started at http://127.2.123.62:38225/ using document root <none> and password file <none>
I20260928 02:44:02.289507 17332 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:02.289577 17332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:02.293788 17332 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:44:02.296185 17347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:02.297555 17332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:44:02.297684 17332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "692fc8a19f074f9c83a0197417fe5a96"
format_stamp: "Formatted at 2026-09-28 02:43:59 on dist-test-slave-rqvb"
server_key: "8b2b8d922f1e24ea8acdcb274fb64909"
server_key_iv: "6172a777a47a3e70bd42229c47cc104a"
server_key_version: "encryptionkey@0"
I20260928 02:44:02.298357 17332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:02.314977 17332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:02.318444 17332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:02.318638 17332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:02.326027 17332 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:40621
I20260928 02:44:02.326319 17399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:40621 every 8 connection(s)
I20260928 02:44:02.327342 17332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:02.331990 17400 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:44:02.333750 17400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Bootstrap starting.
I20260928 02:44:02.334847 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 17332
I20260928 02:44:02.335268 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:34991
--local_ip_for_outbound_sockets=127.2.123.1
--tserver_master_addrs=127.2.123.62:40621
--webserver_port=41447
--webserver_interface=127.2.123.1
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:02.353276 17400 log.cc:824] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:02.364531 17400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:02.364904 17400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96: Bootstrap complete.
I20260928 02:44:02.368799 17400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:44:02.369197 17400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692fc8a19f074f9c83a0197417fe5a96, State: Initialized, Role: FOLLOWER
I20260928 02:44:02.369704 17400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:44:02.369828 17400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:02.369892 17400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:02.370005 17400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:44:02.372018 17400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:44:02.372345 17400 leader_election.cc:302] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692fc8a19f074f9c83a0197417fe5a96; no voters:
I20260928 02:44:02.372644 17400 leader_election.cc:288] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:44:02.372792 17404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:44:02.373061 17404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [term 3 LEADER]: Becoming Leader. State: Replica: 692fc8a19f074f9c83a0197417fe5a96, State: Running, Role: LEADER
I20260928 02:44:02.373385 17404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } }
I20260928 02:44:02.373842 17400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:02.374823 17405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "692fc8a19f074f9c83a0197417fe5a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } } }
I20260928 02:44:02.374953 17405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:02.375244 17406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692fc8a19f074f9c83a0197417fe5a96. Latest consensus state: current_term: 3 leader_uuid: "692fc8a19f074f9c83a0197417fe5a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692fc8a19f074f9c83a0197417fe5a96" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 40621 } } }
I20260928 02:44:02.375334 17406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692fc8a19f074f9c83a0197417fe5a96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:02.375975 17413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:02.381822 17413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:02.382503 17413 catalog_manager.cc:1259] Loaded cluster ID: 688465d0ea3649919a5ab57e6753a91e
I20260928 02:44:02.382565 17413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:44:02.383554 17413 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe6a18ea9ad google::LogMessage::Fail() at ??:0
@ 0x7fe6a18ed88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe6a18ea4de google::LogMessage::Flush() at ??:0
@ 0x7fe6a18eba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe6ac229fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe6ac22ad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe6ac227579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe6ac26b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe6a3b3c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe6a3b3cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe6a3b3e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe6a3b2d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe6a51716db start_thread at ??:0
@ 0x7fe6a03e861f clone at ??:0
W20260928 02:44:02.475906 17402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:02.476188 17402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:02.476234 17402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:02.480652 17402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:02.480738 17402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:02.480825 17402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:44:02.486946 17402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:34991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=41447
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:02.488081 17402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:02.489092 17402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:02.498756 17444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:02.498805 17441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:02.498915 17442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:02.499105 17402 server_base.cc:1034] running on GCE node
I20260928 02:44:02.499640 17402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:02.500449 17402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:02.501613 17402 hybrid_clock.cc:648] HybridClock initialized: now 1790563442501587 us; error 41 us; skew 500 ppm
Sep 28 02:44:02 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:02.504653 17402 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:44:02.505817 17402 webserver.cc:460] Webserver started at http://127.2.123.1:41447/ using document root <none> and password file <none>
I20260928 02:44:02.506578 17402 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:02.506671 17402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:02.511894 17402 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:44:02.514678 17451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:02.516223 17402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:44:02.516350 17402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "349efd8e11884710b85bbb42e632add1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "d87e645cd2d04af52731449bf31f819e"
server_key_iv: "5e59f9a89d4ec6dfc3b4d8df558d5e0c"
server_key_version: "encryptionkey@0"
I20260928 02:44:02.517082 17402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:02.540681 17402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:02.544624 17402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:02.544829 17402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:02.545549 17402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:02.547060 17458 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:44:02.551245 17402 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:44:02.551329 17402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 02:44:02.551397 17402 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:44:02.553594 17402 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:44:02.553643 17402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 02:44:02.554015 17458 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Bootstrap starting.
I20260928 02:44:02.564929 17402 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:34991
I20260928 02:44:02.565016 17565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:34991 every 8 connection(s)
I20260928 02:44:02.566278 17402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:02.572446 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 17402
I20260928 02:44:02.572964 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:41921
--local_ip_for_outbound_sockets=127.2.123.2
--tserver_master_addrs=127.2.123.62:40621
--webserver_port=41371
--webserver_interface=127.2.123.2
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:02.573393 17458 log.cc:824] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:02.580847 17458 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:02.581185 17458 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Bootstrap complete.
I20260928 02:44:02.582145 17458 ts_tablet_manager.cc:1397] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.007s
I20260928 02:44:02.584621 17458 raft_consensus.cc:348] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:02.585039 17458 raft_consensus.cc:729] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349efd8e11884710b85bbb42e632add1, State: Initialized, Role: FOLLOWER
I20260928 02:44:02.585528 17458 consensus_queue.cc:260] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:02.586514 17458 ts_tablet_manager.cc:1428] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
W20260928 02:44:02.622225 17568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:02.568760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:55873)
0928 02:44:02.569379 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.569457 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:02.570650 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:02.621346 (+ 50696us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":161,"threads_started":1}
W20260928 02:44:02.622879 17566 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:40621 (0 consecutive failures): Network error: Failed to ping master at 127.2.123.62:40621: Client connection negotiation failed: client connection to 127.2.123.62:40621: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:44:02.623663 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.623399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:48571)
0928 02:44:02.623534 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.623604 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:44:02.624306 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.624071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:43729)
0928 02:44:02.624169 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.624215 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:44:02.624447 17566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:44:02.703553 17570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:02.703869 17570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:02.703934 17570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:02.708420 17570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:02.708595 17570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:02.708738 17570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:44:02.714663 17570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:41921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=41371
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:02.715816 17570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:02.716869 17570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:02.726963 17577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:02.726980 17576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:02.726963 17579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:02.727356 17570 server_base.cc:1034] running on GCE node
I20260928 02:44:02.727767 17570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:02.728462 17570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:02.729676 17570 hybrid_clock.cc:648] HybridClock initialized: now 1790563442729628 us; error 60 us; skew 500 ppm
Sep 28 02:44:02 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:02.732547 17570 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:44:02.733546 17570 webserver.cc:460] Webserver started at http://127.2.123.2:41371/ using document root <none> and password file <none>
I20260928 02:44:02.734154 17570 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:02.734226 17570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:02.738246 17570 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:44:02.740586 17586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:02.741945 17570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:44:02.742213 17570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "c325a449b4cf4e8c8373333962c734c4"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "b71aa98ef4115a4722ae5244e426a8f0"
server_key_iv: "70d348c4e578c39e9f5863812af8ff72"
server_key_version: "encryptionkey@0"
I20260928 02:44:02.742780 17570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:02.764237 17570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:02.767464 17570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:02.767660 17570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:02.768285 17570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:02.769366 17593 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:44:02.773571 17570 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:44:02.773638 17570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 02:44:02.773703 17570 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:44:02.775414 17570 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:44:02.775460 17570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 02:44:02.775717 17593 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Bootstrap starting.
I20260928 02:44:02.785212 17570 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:41921
I20260928 02:44:02.785254 17700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:41921 every 8 connection(s)
I20260928 02:44:02.786212 17570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:02.786751 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 17570
I20260928 02:44:02.787190 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:46063
--local_ip_for_outbound_sockets=127.2.123.3
--tserver_master_addrs=127.2.123.62:40621
--webserver_port=43501
--webserver_interface=127.2.123.3
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:02.790417 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.788892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:47413)
0928 02:44:02.789428 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.789577 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":147,"threads_started":1}
W20260928 02:44:02.790820 17701 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:40621 (0 consecutive failures): Network error: Failed to ping master at 127.2.123.62:40621: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
I20260928 02:44:02.791414 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.791213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:44733)
0928 02:44:02.791315 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.791360 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:44:02.791976 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.791799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:39731)
0928 02:44:02.791886 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.791928 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:44:02.792119 17701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:44:02.795976 17593 log.cc:824] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:02.804591 17593 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:02.805009 17593 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Bootstrap complete.
I20260928 02:44:02.806107 17593 ts_tablet_manager.cc:1397] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.007s
I20260928 02:44:02.808867 17593 raft_consensus.cc:348] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:02.809302 17593 raft_consensus.cc:729] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c325a449b4cf4e8c8373333962c734c4, State: Initialized, Role: FOLLOWER
I20260928 02:44:02.809772 17593 consensus_queue.cc:260] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:02.810806 17593 ts_tablet_manager.cc:1428] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:44:02.811120 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.810802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:44275)
0928 02:44:02.810960 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.811031 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:44:02.811781 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.811567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:58363)
0928 02:44:02.811668 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.811719 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:44:02.901093 17703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:02.901366 17703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:02.901410 17703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:02.905445 17703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:02.905507 17703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:02.905567 17703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:44:02.910907 17703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:46063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=43501
--tserver_master_addrs=127.2.123.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:02.911927 17703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:02.912900 17703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:02.922472 17711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:02.922456 17714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:02.922474 17712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:02.923092 17703 server_base.cc:1034] running on GCE node
I20260928 02:44:02.923512 17703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:02.924279 17703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:02.925470 17703 hybrid_clock.cc:648] HybridClock initialized: now 1790563442925444 us; error 37 us; skew 500 ppm
Sep 28 02:44:02 dist-test-slave-rqvb krb5kdc[16256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:02.928422 17703 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:44:02.929358 17703 webserver.cc:460] Webserver started at http://127.2.123.3:43501/ using document root <none> and password file <none>
I20260928 02:44:02.929955 17703 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:02.930060 17703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:02.933933 17703 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:44:02.936074 17721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:02.937350 17703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:44:02.937451 17703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "d74e56834e8740c1ac85a56a596ab2e1"
format_stamp: "Formatted at 2026-09-28 02:44:00 on dist-test-slave-rqvb"
server_key: "2d3b32091b8bbe0329d50cc253f81489"
server_key_iv: "275c8b20ea9a539998a423ae3cb8300d"
server_key_version: "encryptionkey@0"
I20260928 02:44:02.938004 17703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:02.958751 17703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:02.961725 17703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:02.961884 17703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:02.962495 17703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:02.963327 17703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:02.963371 17703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:02.963409 17703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:02.963429 17703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:02.972199 17703 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:46063
I20260928 02:44:02.972254 17834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:46063 every 8 connection(s)
I20260928 02:44:02.973210 17703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:02.974872 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 17703
I20260928 02:44:02.977412 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.975595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:56997)
0928 02:44:02.976116 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.976657 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":128,"threads_started":1}
W20260928 02:44:02.977872 17835 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:40621 (0 consecutive failures): Network error: Failed to ping master at 127.2.123.62:40621: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
I20260928 02:44:02.978554 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.978289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:46147)
0928 02:44:02.978425 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.978486 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 02:44:02.979225 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.978979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:37589)
0928 02:44:02.979130 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.979174 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:44:02.979382 17835 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:44:03.625619 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.625070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:42877)
0928 02:44:03.625421 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.625494 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260928 02:44:03.813014 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.812534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:47883)
0928 02:44:03.812844 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.812910 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":81,"threads_started":1}
I20260928 02:44:03.908370 17841 raft_consensus.cc:482] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:03.908591 17841 raft_consensus.cc:504] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:03.909734 17841 leader_election.cc:288] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063), c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921)
Sep 28 02:44:03 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.2@KRBTEST.COM
Sep 28 02:44:03 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.3@KRBTEST.COM
I20260928 02:44:03.923595 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.910232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:46985 (local address 127.2.123.2:41921)
0928 02:44:03.910604 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:03.910617 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:03.910865 (+ 248us) server_negotiation.cc:407] Connection header received
0928 02:44:03.911089 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:03.911094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:03.911435 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:03.911708 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:03.912925 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:03.913866 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:03.915073 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:03.915568 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:03.915714 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:03.918601 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:03.918607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:03.918622 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:03.918658 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:03.920734 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:03.921387 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:03.921392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:03.921397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:03.921567 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:03.922001 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:03.922004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:03.922006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:03.922473 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:03.922604 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:03.923160 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 02:44:03.923412 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":84,"threads_started":1}
I20260928 02:44:03.923592 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.910015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:46063 (local address 127.2.123.1:49771)
0928 02:44:03.910217 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.910234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:03.910340 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:03.911773 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:03.911779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:03.912305 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:03.912764 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:03.912778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:03.914053 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:03.914057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:03.914964 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:03.914973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:03.915682 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:03.915685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:03.915851 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:03.916561 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:03.916583 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:03.918774 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:03.921199 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:03.921202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:03.921204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:03.921407 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:03.921802 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:03.921805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:03.921808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:03.921887 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:03.922588 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:03.922594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:03.922842 (+ 248us) client_negotiation.cc:769] Sending connection context
0928 02:44:03.923003 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 02:44:03.923386 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":135}
I20260928 02:44:03.923624 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.910017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41921 (local address 127.2.123.1:46985)
0928 02:44:03.910758 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.910769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:03.910864 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:03.911772 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:03.911779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:03.912305 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:03.912764 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:03.912778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:03.914028 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:03.914031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:03.914963 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:03.914972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:03.915738 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:03.915742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:03.915859 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:03.916592 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:03.916604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:03.918368 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:03.920884 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:03.920890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:03.920902 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:03.921261 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:03.921709 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:03.921713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:03.921715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:03.921872 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:03.922586 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:03.922592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:03.922844 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 02:44:03.923003 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:44:03.923448 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":640,"spinlock_wait_cycles":11520,"thread_start_us":89,"threads_started":1}
I20260928 02:44:03.923645 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.910142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:49771 (local address 127.2.123.3:46063)
0928 02:44:03.910459 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:03.910476 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:03.910517 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:44:03.911201 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:03.911219 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:03.911515 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:03.911783 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:03.912921 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:03.913914 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:03.915075 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:03.915570 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:03.915713 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:03.919081 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:03.919089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:03.919108 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:03.919148 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:03.921080 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:03.921507 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:03.921513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:03.921518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:03.921709 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:03.922001 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:03.922005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:03.922007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:03.922472 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:03.922598 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:03.923143 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 02:44:03.923439 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
I20260928 02:44:03.925323 17655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62040feea41a4d7992a6843b30f6b911" candidate_uuid: "349efd8e11884710b85bbb42e632add1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c325a449b4cf4e8c8373333962c734c4" is_pre_election: true
I20260928 02:44:03.925668 17655 raft_consensus.cc:2454] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349efd8e11884710b85bbb42e632add1 in term 1.
W20260928 02:44:03.926219 17454 leader_election.cc:341] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063): Not found: Tablet not found: 62040feea41a4d7992a6843b30f6b911
I20260928 02:44:03.926538 17453 leader_election.cc:302] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349efd8e11884710b85bbb42e632add1, c325a449b4cf4e8c8373333962c734c4; no voters: d74e56834e8740c1ac85a56a596ab2e1
I20260928 02:44:03.926836 17841 raft_consensus.cc:2781] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:44:03.927035 17841 raft_consensus.cc:482] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:03.927111 17841 raft_consensus.cc:3037] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:44:03.929425 17841 raft_consensus.cc:504] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:03.929803 17841 leader_election.cc:288] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [CANDIDATE]: Term 2 election: Requested vote from peers d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063), c325a449b4cf4e8c8373333962c734c4 (127.2.123.2:41921)
I20260928 02:44:03.930222 17655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62040feea41a4d7992a6843b30f6b911" candidate_uuid: "349efd8e11884710b85bbb42e632add1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c325a449b4cf4e8c8373333962c734c4"
I20260928 02:44:03.930389 17655 raft_consensus.cc:3037] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 1 FOLLOWER]: Advancing to term 2
W20260928 02:44:03.930480 17454 leader_election.cc:341] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063): Not found: Tablet not found: 62040feea41a4d7992a6843b30f6b911
I20260928 02:44:03.932585 17655 raft_consensus.cc:2454] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349efd8e11884710b85bbb42e632add1 in term 2.
I20260928 02:44:03.933025 17453 leader_election.cc:302] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 349efd8e11884710b85bbb42e632add1, c325a449b4cf4e8c8373333962c734c4; no voters: d74e56834e8740c1ac85a56a596ab2e1
I20260928 02:44:03.933229 17841 raft_consensus.cc:2781] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:44:03.933411 17841 raft_consensus.cc:686] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [term 2 LEADER]: Becoming Leader. State: Replica: 349efd8e11884710b85bbb42e632add1, State: Running, Role: LEADER
I20260928 02:44:03.933777 17841 consensus_queue.cc:237] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:03.934265 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.933882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:46323)
0928 02:44:03.934128 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.934188 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:44:03.980703 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.980157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:50885)
0928 02:44:03.980502 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.980584 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
W20260928 02:44:04.379675 17454 consensus_peers.cc:487] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 -> Peer d74e56834e8740c1ac85a56a596ab2e1 (127.2.123.3:46063): Couldn't send request to peer d74e56834e8740c1ac85a56a596ab2e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62040feea41a4d7992a6843b30f6b911. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:44:04.402232 17655 raft_consensus.cc:1261] T 62040feea41a4d7992a6843b30f6b911 P c325a449b4cf4e8c8373333962c734c4 [term 2 FOLLOWER]: Refusing update from remote peer 349efd8e11884710b85bbb42e632add1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 02:44:04.402994 17841 consensus_queue.cc:1035] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 02:44:04.405243 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.404844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:58201)
0928 02:44:04.405052 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.405119 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260928 02:44:04.407636 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.407375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:47149)
0928 02:44:04.407491 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.407551 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:44:04.408061 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.407842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:45551)
0928 02:44:04.407930 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.407979 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 02:44:04.953140 17853 ts_tablet_manager.cc:927] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Initiating tablet copy from peer 349efd8e11884710b85bbb42e632add1 (127.2.123.1:34991)
I20260928 02:44:04.953724 17853 tablet_copy_client.cc:316] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: tablet copy: Beginning tablet copy session from remote peer at address 127.2.123.1:34991
Sep 28 02:44:04 dist-test-slave-rqvb krb5kdc[16256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:44:04.963730 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.954261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:34991 (local address 127.2.123.3:54447)
0928 02:44:04.954519 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.954544 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:04.954683 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:04.955157 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:04.955163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:04.955542 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:04.955669 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:04.955676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:04.956670 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:04.956673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:04.957373 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:04.957383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:04.958118 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:04.958121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:04.958204 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:04.958735 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:04.958747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:04.960148 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:04.961919 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:04.961927 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:04.961932 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:04.962194 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:04.962513 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:04.962516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:04.962518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:04.962608 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:04.963119 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:04.963125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:04.963261 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:44:04.963415 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:44:04.963597 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260928 02:44:04.963836 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.954411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:54447 (local address 127.2.123.1:34991)
0928 02:44:04.954679 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:04.954685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:04.954709 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:44:04.954774 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:04.954778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:04.954965 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:04.955093 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:04.955799 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:04.956558 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:04.957483 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:04.957985 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:04.958103 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:04.960286 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:04.960295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:04.960306 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:04.960328 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:04.961752 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:04.962311 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:04.962316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:04.962317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:04.962404 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:04.962696 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:04.962699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:04.962700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:04.963009 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:04.963119 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:04.963558 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 02:44:04.963695 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260928 02:44:04.964809 17540 tablet_copy_service.cc:140] P 349efd8e11884710b85bbb42e632add1: Received BeginTabletCopySession request for tablet 62040feea41a4d7992a6843b30f6b911 from peer d74e56834e8740c1ac85a56a596ab2e1 ({username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:54447)
I20260928 02:44:04.964962 17540 tablet_copy_service.cc:161] P 349efd8e11884710b85bbb42e632add1: Beginning new tablet copy session on tablet 62040feea41a4d7992a6843b30f6b911 from peer d74e56834e8740c1ac85a56a596ab2e1 at {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:54447: session id = d74e56834e8740c1ac85a56a596ab2e1-62040feea41a4d7992a6843b30f6b911
I20260928 02:44:04.966235 17540 tablet_copy_source_session.cc:205] T 62040feea41a4d7992a6843b30f6b911 P 349efd8e11884710b85bbb42e632add1: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:44:04.970875 17853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62040feea41a4d7992a6843b30f6b911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:04.978719 17853 tablet_copy_client.cc:791] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: tablet copy: Starting download of 0 data blocks...
I20260928 02:44:04.978932 17853 tablet_copy_client.cc:655] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: tablet copy: Starting download of 1 WAL segments...
I20260928 02:44:04.980755 17853 tablet_copy_client.cc:531] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:44:04.981628 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.981385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:55601)
0928 02:44:04.981511 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.981562 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:44:04.982659 17853 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Bootstrap starting.
I20260928 02:44:04.988077 17853 log.cc:824] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:04.994691 17853 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:04.994989 17853 tablet_bootstrap.cc:492] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Bootstrap complete.
I20260928 02:44:04.995302 17853 ts_tablet_manager.cc:1397] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260928 02:44:04.997797 17853 raft_consensus.cc:348] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:04.998088 17853 raft_consensus.cc:729] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74e56834e8740c1ac85a56a596ab2e1, State: Initialized, Role: FOLLOWER
I20260928 02:44:04.998545 17853 consensus_queue.cc:260] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e56834e8740c1ac85a56a596ab2e1" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 46063 } } peers { permanent_uuid: "c325a449b4cf4e8c8373333962c734c4" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41921 } } peers { permanent_uuid: "349efd8e11884710b85bbb42e632add1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 34991 } }
I20260928 02:44:04.999496 17853 ts_tablet_manager.cc:1428] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 02:44:04.999822 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.999572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:51781)
0928 02:44:04.999694 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.999750 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:44:05.000197 17540 tablet_copy_service.cc:342] P 349efd8e11884710b85bbb42e632add1: Request end of tablet copy session d74e56834e8740c1ac85a56a596ab2e1-62040feea41a4d7992a6843b30f6b911 received from {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:54447
I20260928 02:44:05.000300 17540 tablet_copy_service.cc:433] P 349efd8e11884710b85bbb42e632add1: ending tablet copy session d74e56834e8740c1ac85a56a596ab2e1-62040feea41a4d7992a6843b30f6b911 on tablet 62040feea41a4d7992a6843b30f6b911 with peer d74e56834e8740c1ac85a56a596ab2e1
I20260928 02:44:05.000434 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:05.000213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:42715)
0928 02:44:05.000321 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:05.000370 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:44:05.408921 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:05.408414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:59073)
0928 02:44:05.408659 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:05.408732 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260928 02:44:05.409286 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:05.408737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:47275)
0928 02:44:05.408942 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:05.409199 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":57,"threads_started":1}
I20260928 02:44:05.409691 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:05.409462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:38045)
0928 02:44:05.409575 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:05.409627 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 02:44:05.409902 17789 raft_consensus.cc:1203] T 62040feea41a4d7992a6843b30f6b911 P d74e56834e8740c1ac85a56a596ab2e1 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 02:44:05.410795 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:05.410544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:52097)
0928 02:44:05.410674 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:05.410725 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:44:06.410583 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:06.410118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:44071)
0928 02:44:06.410415 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:06.410474 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":71,"threads_started":1}
I20260928 02:44:06.410735 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:06.410348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:51281)
0928 02:44:06.410576 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:06.410632 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260928 02:44:06.411923 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:06.411463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:48923)
0928 02:44:06.411757 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:06.411811 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260928 02:44:07.411867 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:07.411458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:37807)
0928 02:44:07.411707 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:07.411763 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260928 02:44:07.411882 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:07.411458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:53679)
0928 02:44:07.411744 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:07.411798 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":70,"threads_started":1}
I20260928 02:44:07.412990 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:07.412621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:34129)
0928 02:44:07.412852 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:07.412906 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260928 02:44:08.413344 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:08.412859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:59305)
0928 02:44:08.413160 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:08.413219 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260928 02:44:08.413347 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:08.412894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:36253)
0928 02:44:08.413159 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:08.413217 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260928 02:44:08.414265 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:08.413857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:34783)
0928 02:44:08.414121 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:08.414175 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260928 02:44:08.414276 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:08.414019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:54445)
0928 02:44:08.414121 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:08.414174 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:44:09.414804 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:09.414276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:55811)
0928 02:44:09.414519 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:09.414668 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260928 02:44:09.415304 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:09.414898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:57689)
0928 02:44:09.415167 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:09.415224 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260928 02:44:09.415391 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:09.414967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:56979)
0928 02:44:09.415263 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:09.415309 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":60,"threads_started":1}
I20260928 02:44:10.416348 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:10.415871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:54401)
0928 02:44:10.416151 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:10.416223 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
I20260928 02:44:10.416496 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:10.416022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:35383)
0928 02:44:10.416362 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:10.416409 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260928 02:44:10.416764 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:10.416024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:38417)
0928 02:44:10.416279 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:10.416351 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260928 02:44:11.417761 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:11.417272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:34155)
0928 02:44:11.417570 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:11.417645 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260928 02:44:11.417786 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:11.417272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:55173)
0928 02:44:11.417623 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:11.417685 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":77,"threads_started":1}
I20260928 02:44:11.417786 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:11.417419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:39887)
0928 02:44:11.417645 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:11.417697 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260928 02:44:12.419107 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:12.418606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:57337)
0928 02:44:12.418970 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:12.419023 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":71,"threads_started":1}
I20260928 02:44:12.419111 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:12.418612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:41653)
0928 02:44:12.418959 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:12.419009 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":79,"threads_started":1}
I20260928 02:44:12.419111 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:12.418612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:48407)
0928 02:44:12.418903 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:12.418999 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260928 02:44:13.420569 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:13.420057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:43251)
0928 02:44:13.420390 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:13.420458 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":71,"threads_started":1}
I20260928 02:44:13.420569 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:13.420098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:55153)
0928 02:44:13.420396 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:13.420458 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":56,"threads_started":1}
I20260928 02:44:13.420763 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:13.420035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:41695)
0928 02:44:13.420485 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:13.420648 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":153,"threads_started":1}
I20260928 02:44:14.421707 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:14.421292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:34429)
0928 02:44:14.421527 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:14.421591 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260928 02:44:14.421766 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:14.421289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:54053)
0928 02:44:14.421634 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:14.421685 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":80,"threads_started":1}
I20260928 02:44:14.421766 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:14.421372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:49203)
0928 02:44:14.421609 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:14.421667 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":56,"threads_started":1}
I20260928 02:44:15.423027 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:15.422520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:39801)
0928 02:44:15.422840 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:15.422907 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
I20260928 02:44:15.423144 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:15.422732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:35315)
0928 02:44:15.422999 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:15.423053 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":58,"threads_started":1}
I20260928 02:44:15.423082 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:15.422647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:36933)
0928 02:44:15.422906 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:15.422971 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":67,"threads_started":1}
I20260928 02:44:16.424321 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:16.423862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:51047)
0928 02:44:16.424138 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:16.424204 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260928 02:44:16.424345 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:16.423880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:34591)
0928 02:44:16.424217 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:16.424267 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":75,"threads_started":1}
I20260928 02:44:16.424352 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:16.423878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:54903)
0928 02:44:16.424203 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:16.424261 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":82,"threads_started":1}
I20260928 02:44:17.425536 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:17.425092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.3:58155)
0928 02:44:17.425353 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:17.425417 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260928 02:44:17.425576 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:17.425191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.1:47739)
0928 02:44:17.425431 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:17.425493 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260928 02:44:17.425841 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:17.425321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:40621 (local address 127.2.123.2:38151)
0928 02:44:17.425538 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:17.425588 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":63,"threads_started":1}
I20260928 02:44:17.982761 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 17402
I20260928 02:44:17.987694 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 17570
I20260928 02:44:17.992497 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 17703
2026-09-28T02:44:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19342 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:44:18 dist-test-slave-rqvb krb5kdc[17906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:44:18 dist-test-slave-rqvb krb5kdc[17906](info): set up 2 sockets
Sep 28 02:44:18 dist-test-slave-rqvb krb5kdc[17906](info): commencing operation
krb5kdc: starting...
W20260928 02:44:18.970340 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:44:19.094134 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41415
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:41415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:19.204442 17919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:19.204743 17919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:19.204802 17919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:19.208961 17919 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:44:19.209034 17919 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:44:19.209054 17919 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:44:19.209072 17919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:19.209095 17919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:19.209120 17919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:19.209148 17919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:19.209172 17919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:19.214673 17919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:41415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:19.215795 17919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:19.216861 17919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:19.225796 17925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:19.225811 17924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:19.225811 17927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:19.225864 17919 server_base.cc:1034] running on GCE node
I20260928 02:44:19.226662 17919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:19.227933 17919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:19.229183 17919 hybrid_clock.cc:648] HybridClock initialized: now 1790563459229160 us; error 44 us; skew 500 ppm
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:19.232218 17919 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:44:19.233106 17919 webserver.cc:460] Webserver started at http://127.2.123.62:38953/ using document root <none> and password file <none>
I20260928 02:44:19.233640 17919 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:19.233696 17919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:19.233855 17919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:19.236090 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "38c3448b1a0944a389bf7df4419b2feb"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "8ae8877a42dec5eaab740c5e5a8e30bd"
server_key_iv: "2ef0decc112e008581306e34bf35f57b"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.236620 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "38c3448b1a0944a389bf7df4419b2feb"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "8ae8877a42dec5eaab740c5e5a8e30bd"
server_key_iv: "2ef0decc112e008581306e34bf35f57b"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.240386 17919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:44:19.242830 17934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.244131 17919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:44:19.244243 17919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "38c3448b1a0944a389bf7df4419b2feb"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "8ae8877a42dec5eaab740c5e5a8e30bd"
server_key_iv: "2ef0decc112e008581306e34bf35f57b"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.244333 17919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:19.285851 17919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:19.289230 17919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:19.289403 17919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:19.296154 17919 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:41415
I20260928 02:44:19.296419 17986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:41415 every 8 connection(s)
I20260928 02:44:19.297119 17919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:19.300002 17987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.301091 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 17919
I20260928 02:44:19.301211 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:44:19.301571 2540 external_mini_cluster.cc:1428] Setting key a0c2ad5068f4efc0815e267470a41a97
I20260928 02:44:19.306500 17987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Bootstrap starting.
I20260928 02:44:19.309260 17987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.310192 17987 log.cc:824] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:19.312449 17987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: No bootstrap required, opened a new log
I20260928 02:44:19.316228 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.303005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.2.123.62:41415)
0928 02:44:19.303491 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.303508 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.303545 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:44:19.304209 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.304227 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.304582 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.304865 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:19.305728 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.306815 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.307965 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.308455 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.308592 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.311431 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:19.311436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:19.311451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:19.311479 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:19.313383 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.313843 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.313848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.313853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.314001 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.314266 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.314269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.314271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.314697 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:19.314898 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.315154 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.315463 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":138,"threads_started":1}
I20260928 02:44:19.316160 17987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:44:19.316390 17987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.316442 17987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c3448b1a0944a389bf7df4419b2feb, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.316934 17987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:44:19.317063 17987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:19.317118 17987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:19.317212 17987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.318415 17987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:44:19.318764 17987 leader_election.cc:302] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38c3448b1a0944a389bf7df4419b2feb; no voters:
I20260928 02:44:19.319077 17987 leader_election.cc:288] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:44:19.319204 17992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:19.319448 17992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 1 LEADER]: Becoming Leader. State: Replica: 38c3448b1a0944a389bf7df4419b2feb, State: Running, Role: LEADER
I20260928 02:44:19.319739 17992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:44:19.320281 17987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:19.321498 17994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38c3448b1a0944a389bf7df4419b2feb. Latest consensus state: current_term: 1 leader_uuid: "38c3448b1a0944a389bf7df4419b2feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } } }
I20260928 02:44:19.321501 17993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38c3448b1a0944a389bf7df4419b2feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } } }
I20260928 02:44:19.321651 17993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:19.321862 17994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:19.322057 18002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:19.324730 18002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:19.330286 18002 catalog_manager.cc:1347] Generated new cluster ID: 9606a57387234635ac01da8685b8a321
I20260928 02:44:19.330372 18002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:44:19.342430 18002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:44:19.343552 18002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:19.354978 18002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Generated new TSK 0
I20260928 02:44:19.355696 18002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:44:19.384032 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41415
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:19.492466 18013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:19.492731 18013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:19.492775 18013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:19.496776 18013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:19.496850 18013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:19.496910 18013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:44:19.502291 18013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:19.503341 18013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:19.504372 18013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:19.513886 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:19.513871 18021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:19.513871 18019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:19.514146 18013 server_base.cc:1034] running on GCE node
I20260928 02:44:19.514569 18013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:19.515239 18013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:19.516435 18013 hybrid_clock.cc:648] HybridClock initialized: now 1790563459516411 us; error 34 us; skew 500 ppm
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:19.519150 18013 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:44:19.520000 18013 webserver.cc:460] Webserver started at http://127.2.123.1:37275/ using document root <none> and password file <none>
I20260928 02:44:19.520568 18013 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:19.520639 18013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:19.520835 18013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:19.522890 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "758117e6d8c3431280e882c8a28ebc2e"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "11d3be75389bfaa3df7f2d5c3367afb3"
server_key_iv: "385a79305f13a8fa9eecf0aeb775bec8"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.523420 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "758117e6d8c3431280e882c8a28ebc2e"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "11d3be75389bfaa3df7f2d5c3367afb3"
server_key_iv: "385a79305f13a8fa9eecf0aeb775bec8"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.527096 18013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:44:19.529541 18028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.530893 18013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:44:19.531049 18013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "758117e6d8c3431280e882c8a28ebc2e"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "11d3be75389bfaa3df7f2d5c3367afb3"
server_key_iv: "385a79305f13a8fa9eecf0aeb775bec8"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.531147 18013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:19.543843 18013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:19.546818 18013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:19.547037 18013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:19.547614 18013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:19.548399 18013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:19.548446 18013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.548496 18013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:19.548527 18013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.556874 18013 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:44653
I20260928 02:44:19.556998 18141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:44653 every 8 connection(s)
I20260928 02:44:19.557797 18013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:19.559917 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 18013
I20260928 02:44:19.560053 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:44:19.560343 2540 external_mini_cluster.cc:1428] Setting key 3bf9945f12b1d089f5550776194d8599
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:44:19.572839 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.559888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:36891 (local address 127.2.123.62:41415)
0928 02:44:19.560047 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.560052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.561005 (+ 953us) server_negotiation.cc:407] Connection header received
0928 02:44:19.562375 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.562378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.562431 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.562506 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:19.564098 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.564657 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.566369 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.566617 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.566701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.569434 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:19.569437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:19.569439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:19.569464 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:19.570775 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.571261 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.571264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.571265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.571337 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.571672 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.571676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.571678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.571894 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:19.572048 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.572535 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.572677 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:44:19.573642 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.560384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:36891)
0928 02:44:19.560867 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.560921 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.562174 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.562696 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.562703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.563215 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:19.563930 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.563952 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.564789 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.564792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.566237 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.566250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.566778 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.566782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.566997 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.567646 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:19.567667 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:19.569275 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:19.570881 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:19.570886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:19.570899 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:19.571192 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:19.571406 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:19.571408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:19.571409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:19.571581 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:19.572052 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:19.572057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:19.572307 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 02:44:19.572478 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 02:44:19.572741 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260928 02:44:19.574837 18142 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41415
I20260928 02:44:19.575101 18142 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:19.575709 18142 heartbeater.cc:511] Master 127.2.123.62:41415 requested a full tablet report, sending...
I20260928 02:44:19.577342 17950 ts_manager.cc:194] Registered new tserver with Master: 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653)
I20260928 02:44:19.578519 17950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:36891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:44:19.588402 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41415
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:19.695497 18147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:19.695758 18147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:19.695799 18147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:19.699951 18147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:19.700017 18147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:19.700078 18147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:44:19.705399 18147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:19.706537 18147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:19.707551 18147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:19.716809 18153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:19.716809 18155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:19.716893 18147 server_base.cc:1034] running on GCE node
W20260928 02:44:19.716835 18152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:19.717492 18147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:19.718202 18147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:19.719379 18147 hybrid_clock.cc:648] HybridClock initialized: now 1790563459719343 us; error 48 us; skew 500 ppm
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:19.722185 18147 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:44:19.722999 18147 webserver.cc:460] Webserver started at http://127.2.123.2:42799/ using document root <none> and password file <none>
I20260928 02:44:19.723570 18147 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:19.723634 18147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:19.723834 18147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:19.725806 18147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "e1a3180e712a4349841514cf01272c26"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "f568725d75fd9b60264351c9fb428276"
server_key_iv: "3b4e0cfb9ac1a60b0ef26394e187598d"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.726346 18147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "e1a3180e712a4349841514cf01272c26"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "f568725d75fd9b60264351c9fb428276"
server_key_iv: "3b4e0cfb9ac1a60b0ef26394e187598d"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.729862 18147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:44:19.732261 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.733530 18147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:44:19.733623 18147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "e1a3180e712a4349841514cf01272c26"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "f568725d75fd9b60264351c9fb428276"
server_key_iv: "3b4e0cfb9ac1a60b0ef26394e187598d"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.733755 18147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:19.766680 18147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:19.770078 18147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:19.770231 18147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:19.770777 18147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:19.771576 18147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:19.771629 18147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.771667 18147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:19.771687 18147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.779855 18147 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:34729
I20260928 02:44:19.779871 18275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:34729 every 8 connection(s)
I20260928 02:44:19.780740 18147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:19.784904 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 18147
I20260928 02:44:19.785028 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:44:19.785382 2540 external_mini_cluster.cc:1428] Setting key df4258775fd7b14a0c697be3d168a85c
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:44:19.794647 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.782503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:49415 (local address 127.2.123.62:41415)
0928 02:44:19.782669 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.782674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.783458 (+ 784us) server_negotiation.cc:407] Connection header received
0928 02:44:19.784670 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.784673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.784720 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.784789 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:19.786301 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.787007 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.788542 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.788864 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.788972 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.791376 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:19.791379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:19.791381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:19.791418 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:19.792676 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.793155 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.793158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.793159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.793235 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.793545 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.793548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.793550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.793724 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:19.793806 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.794355 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.794485 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:44:19.795580 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.782887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:49415)
0928 02:44:19.783313 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.783355 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.784477 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.785039 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.785049 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.785551 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:19.786165 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.786182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.787142 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.787146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.788414 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.788425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.789005 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.789008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.789153 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.789725 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:19.789741 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:19.791218 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:19.792787 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:19.792793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:19.792809 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:19.793062 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:19.793316 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:19.793319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:19.793321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:19.793459 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:19.793819 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:19.793826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:19.794129 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 02:44:19.794301 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 02:44:19.794632 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:44:19.796859 18276 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41415
I20260928 02:44:19.797130 18276 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:19.797646 18276 heartbeater.cc:511] Master 127.2.123.62:41415 requested a full tablet report, sending...
I20260928 02:44:19.798816 17950 ts_manager.cc:194] Registered new tserver with Master: e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729)
I20260928 02:44:19.799419 17950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:49415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:44:19.812928 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41415
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:19.923207 18281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:19.923477 18281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:19.923519 18281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:19.927506 18281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:19.927563 18281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:19.927623 18281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:44:19.932824 18281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:19.933791 18281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:19.934842 18281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:19.944732 18286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:19.944723 18289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:19.944993 18281 server_base.cc:1034] running on GCE node
W20260928 02:44:19.944725 18287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:19.945425 18281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:19.946069 18281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:19.947237 18281 hybrid_clock.cc:648] HybridClock initialized: now 1790563459947204 us; error 40 us; skew 500 ppm
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:19.950091 18281 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:44:19.951017 18281 webserver.cc:460] Webserver started at http://127.2.123.3:45847/ using document root <none> and password file <none>
I20260928 02:44:19.951556 18281 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:19.951624 18281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:19.951813 18281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:19.953859 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "02ea3b8634c246e6b6f66d5c3ba8322a"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "a7217c00aedaaa51b9d9a90b76f2c21e"
server_key_iv: "472c58ab463f6f58a6b8b9ad65f35cdd"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.954420 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "02ea3b8634c246e6b6f66d5c3ba8322a"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "a7217c00aedaaa51b9d9a90b76f2c21e"
server_key_iv: "472c58ab463f6f58a6b8b9ad65f35cdd"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.958020 18281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:44:19.960462 18296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.961923 18281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:44:19.962078 18281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "02ea3b8634c246e6b6f66d5c3ba8322a"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "a7217c00aedaaa51b9d9a90b76f2c21e"
server_key_iv: "472c58ab463f6f58a6b8b9ad65f35cdd"
server_key_version: "encryptionkey@0"
I20260928 02:44:19.962193 18281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:19.973801 18281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:19.976831 18281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:19.977018 18281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:19.977605 18281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:19.978449 18281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:19.978509 18281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.978561 18281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:19.978596 18281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.987171 18281 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:35451
I20260928 02:44:19.987198 18409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:35451 every 8 connection(s)
I20260928 02:44:19.988039 18281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:19.989101 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 18281
I20260928 02:44:19.989197 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:44:19.989452 2540 external_mini_cluster.cc:1428] Setting key 8d0b562a84f0807b93f383215cd8e834
Sep 28 02:44:19 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:20.003345 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.989814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:41575 (local address 127.2.123.62:41415)
0928 02:44:19.990035 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.990040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.990989 (+ 949us) server_negotiation.cc:407] Connection header received
0928 02:44:19.992262 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.992264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.992310 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.992379 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:19.993705 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.994592 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.996285 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.996563 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.996627 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.999618 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:19.999621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:19.999623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:19.999650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:20.001223 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:20.001750 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:20.001753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:20.001754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:20.001835 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:20.002228 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:20.002231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:20.002233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:20.002435 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:20.002536 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:20.003050 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 02:44:20.003172 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:44:20.004174 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.990359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:41575)
0928 02:44:19.990835 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.990887 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.992072 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.992518 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.992524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.992890 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:19.993543 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.993564 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.994730 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.994733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.996137 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.996148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.996801 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.996804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.996961 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.997583 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:19.997604 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:19.999400 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:20.001336 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:20.001341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:20.001355 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:20.001650 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:20.001930 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:20.001934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:20.001936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:20.002137 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:20.002537 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:20.002544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:20.002809 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 02:44:20.002975 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 02:44:20.003335 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260928 02:44:20.005278 18410 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41415
I20260928 02:44:20.005518 18410 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:20.006089 18410 heartbeater.cc:511] Master 127.2.123.62:41415 requested a full tablet report, sending...
I20260928 02:44:20.007115 17950 ts_manager.cc:194] Registered new tserver with Master: 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451)
I20260928 02:44:20.007678 17950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:41575
I20260928 02:44:20.014961 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:44:20.027177 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.016833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.2.123.62:41415)
0928 02:44:20.017038 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:20.017042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:20.017143 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:44:20.017264 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:20.017267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:20.017308 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:20.017373 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:20.018003 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.018531 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.019335 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.019553 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.019631 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.023165 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:20.023168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:20.023170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:20.023197 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:20.025009 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:20.025928 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:20.025931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:20.025932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:20.026058 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:20.026325 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:20.026328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:20.026329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:20.026570 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:20.026670 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:20.026866 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:44:20.027017 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260928 02:44:20.030272 17950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:44:20.033299 17950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:20.048271 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.043005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.2.123.3:35451)
0928 02:44:20.043604 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:20.043615 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:20.043639 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:44:20.043715 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:20.043721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:20.043930 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:20.044091 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:20.044501 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.045249 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.046704 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.047577 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.047668 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.047823 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:20.048011 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:44:20.048117 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":94,"threads_started":1}
I20260928 02:44:20.048420 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.043034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:44653 (local address 127.0.0.1:57924)
0928 02:44:20.043313 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:20.043323 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:20.043386 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:20.044084 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:20.044091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:20.044125 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:20.044430 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.044437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.045921 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.045925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.047036 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.047045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.048049 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.048053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.048143 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.048167 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:44:20.048216 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:44:20.048282 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":140,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260928 02:44:20.048306 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.042363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35451 (local address 127.0.0.1:39072)
0928 02:44:20.043253 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:20.043266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:20.043353 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:20.044164 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:20.044166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:20.044176 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:20.044378 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.044386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.045372 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.045375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.046557 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.046567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.047682 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.047685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.047824 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.047866 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:44:20.047975 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:44:20.048107 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":41,"spinlock_wait_cycles":7936,"thread_start_us":105,"threads_started":1}
I20260928 02:44:20.048797 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.042312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:34729 (local address 127.0.0.1:46806)
0928 02:44:20.042726 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:20.042778 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:20.042915 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:20.044103 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:20.044105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:20.044128 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:20.044428 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.044437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.045892 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.045896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.047342 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.047353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.048404 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.048407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.048497 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.048518 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:44:20.048574 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:44:20.048637 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":80,"spinlock_wait_cycles":4224,"thread_start_us":77,"threads_started":1}
I20260928 02:44:20.048859 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.043203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.2.123.1:44653)
0928 02:44:20.043618 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:20.043636 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:20.043654 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:44:20.043715 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:20.043721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:20.043930 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:20.044091 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:20.044579 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.045775 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.047149 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.047900 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.048421 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.048576 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:20.048625 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:44:20.048723 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":102,"threads_started":1}
I20260928 02:44:20.050424 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.042442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.2.123.2:34729)
0928 02:44:20.043628 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:20.043640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:20.043662 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:44:20.043735 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:20.043739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:20.043953 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:20.044321 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:20.044564 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.045752 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.047466 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.048279 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.049991 (+ 1712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.050144 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:20.050193 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:44:20.050289 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":143,"threads_started":1}
I20260928 02:44:20.051581 18344 tablet_service.cc:1467] Processing CreateTablet for tablet 5ceb51e6fdd545a5be8bce4d87c274b2 (DEFAULT_TABLE table=test-table [id=bf55a669e0374b26bad425e0674a25c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:20.051618 18076 tablet_service.cc:1467] Processing CreateTablet for tablet 5ceb51e6fdd545a5be8bce4d87c274b2 (DEFAULT_TABLE table=test-table [id=bf55a669e0374b26bad425e0674a25c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:20.052543 18344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ceb51e6fdd545a5be8bce4d87c274b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:20.052662 18076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ceb51e6fdd545a5be8bce4d87c274b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:20.052871 18210 tablet_service.cc:1467] Processing CreateTablet for tablet 5ceb51e6fdd545a5be8bce4d87c274b2 (DEFAULT_TABLE table=test-table [id=bf55a669e0374b26bad425e0674a25c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:20.053890 18210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ceb51e6fdd545a5be8bce4d87c274b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:20.058995 18427 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Bootstrap starting.
I20260928 02:44:20.060463 18428 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Bootstrap starting.
I20260928 02:44:20.061301 18429 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Bootstrap starting.
I20260928 02:44:20.061429 18427 tablet_bootstrap.cc:654] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:20.062341 18427 log.cc:824] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:20.062747 18428 tablet_bootstrap.cc:654] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:20.063436 18429 tablet_bootstrap.cc:654] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:20.063866 18428 log.cc:824] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:20.064260 18429 log.cc:824] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:20.064606 18427 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: No bootstrap required, opened a new log
I20260928 02:44:20.064833 18427 ts_tablet_manager.cc:1397] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:44:20.065897 18428 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: No bootstrap required, opened a new log
I20260928 02:44:20.066051 18429 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: No bootstrap required, opened a new log
I20260928 02:44:20.066097 18428 ts_tablet_manager.cc:1397] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:44:20.066219 18429 ts_tablet_manager.cc:1397] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 02:44:20.068835 18427 raft_consensus.cc:348] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.068920 18429 raft_consensus.cc:348] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.069115 18429 raft_consensus.cc:374] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:20.069106 18427 raft_consensus.cc:374] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:20.069146 18429 raft_consensus.cc:729] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a3180e712a4349841514cf01272c26, State: Initialized, Role: FOLLOWER
I20260928 02:44:20.069149 18427 raft_consensus.cc:729] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ea3b8634c246e6b6f66d5c3ba8322a, State: Initialized, Role: FOLLOWER
I20260928 02:44:20.069514 18429 consensus_queue.cc:260] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.069585 18427 consensus_queue.cc:260] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.069777 18428 raft_consensus.cc:348] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.070154 18276 heartbeater.cc:503] Master 127.2.123.62:41415 was elected leader, sending a full tablet report...
I20260928 02:44:20.070456 18429 ts_tablet_manager.cc:1428] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260928 02:44:20.070596 18427 ts_tablet_manager.cc:1428] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:44:20.070919 18410 heartbeater.cc:503] Master 127.2.123.62:41415 was elected leader, sending a full tablet report...
I20260928 02:44:20.071095 18428 raft_consensus.cc:374] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:20.071173 18428 raft_consensus.cc:729] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758117e6d8c3431280e882c8a28ebc2e, State: Initialized, Role: FOLLOWER
I20260928 02:44:20.071574 18428 consensus_queue.cc:260] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.072297 18142 heartbeater.cc:503] Master 127.2.123.62:41415 was elected leader, sending a full tablet report...
I20260928 02:44:20.072449 18428 ts_tablet_manager.cc:1428] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260928 02:44:20.238953 18411 tablet.cc:2369] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:44:20.281891 18277 tablet.cc:2369] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:44:20.309128 18143 tablet.cc:2369] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:44:20.380656 18434 raft_consensus.cc:482] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:20.380893 18434 raft_consensus.cc:504] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.382061 18434 leader_election.cc:288] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729), 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653)
I20260928 02:44:20.385546 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.382387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:34027 (local address 127.2.123.2:34729)
0928 02:44:20.382504 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:20.382509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:20.382528 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:44:20.382628 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:20.382632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:20.382684 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:20.382879 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:20.383145 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.383647 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.384652 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.385038 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.385125 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.385195 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:20.385374 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:44:20.385440 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:44:20.385547 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.382288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:34729 (local address 127.2.123.3:34027)
0928 02:44:20.382411 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:20.382427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:20.382523 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:20.382899 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:20.382903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:20.382921 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:20.383034 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.383040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.383758 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.383762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.384528 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.384536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.385145 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.385149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.385263 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.385281 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:44:20.385345 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:44:20.385412 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260928 02:44:20.385754 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.382561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:44653 (local address 127.2.123.3:60217)
0928 02:44:20.382897 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:20.382908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:20.382995 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:20.383204 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:20.383207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:20.383227 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:20.383341 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.383347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.384042 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.384046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.384786 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:20.384794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.385383 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.385386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:20.385475 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.385496 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:44:20.385558 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:44:20.385619 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":131,"threads_started":1}
I20260928 02:44:20.385761 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.382691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:60217 (local address 127.2.123.1:44653)
0928 02:44:20.382825 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:20.382829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:20.382986 (+ 157us) server_negotiation.cc:407] Connection header received
0928 02:44:20.383080 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:20.383083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:20.383127 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:20.383185 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:20.383444 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.383943 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.384900 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.385279 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.385366 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.385422 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:20.385583 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:44:20.385655 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:44:20.386358 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ceb51e6fdd545a5be8bce4d87c274b2" candidate_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a3180e712a4349841514cf01272c26" is_pre_election: true
I20260928 02:44:20.386531 18096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ceb51e6fdd545a5be8bce4d87c274b2" candidate_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758117e6d8c3431280e882c8a28ebc2e" is_pre_election: true
I20260928 02:44:20.386653 18230 raft_consensus.cc:2454] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ea3b8634c246e6b6f66d5c3ba8322a in term 0.
I20260928 02:44:20.386796 18096 raft_consensus.cc:2454] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ea3b8634c246e6b6f66d5c3ba8322a in term 0.
I20260928 02:44:20.387101 18298 leader_election.cc:302] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ea3b8634c246e6b6f66d5c3ba8322a, e1a3180e712a4349841514cf01272c26; no voters:
I20260928 02:44:20.387353 18434 raft_consensus.cc:2781] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:20.387436 18434 raft_consensus.cc:482] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:20.387472 18434 raft_consensus.cc:3037] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:20.388551 18434 raft_consensus.cc:504] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.388895 18434 leader_election.cc:288] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [CANDIDATE]: Term 1 election: Requested vote from peers e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729), 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653)
I20260928 02:44:20.389297 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ceb51e6fdd545a5be8bce4d87c274b2" candidate_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a3180e712a4349841514cf01272c26"
I20260928 02:44:20.389355 18096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ceb51e6fdd545a5be8bce4d87c274b2" candidate_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758117e6d8c3431280e882c8a28ebc2e"
I20260928 02:44:20.389416 18230 raft_consensus.cc:3037] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:20.389478 18096 raft_consensus.cc:3037] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:20.390487 18230 raft_consensus.cc:2454] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ea3b8634c246e6b6f66d5c3ba8322a in term 1.
I20260928 02:44:20.390585 18096 raft_consensus.cc:2454] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ea3b8634c246e6b6f66d5c3ba8322a in term 1.
I20260928 02:44:20.390823 18298 leader_election.cc:302] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ea3b8634c246e6b6f66d5c3ba8322a, e1a3180e712a4349841514cf01272c26; no voters:
I20260928 02:44:20.391001 18434 raft_consensus.cc:2781] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:20.391251 18434 raft_consensus.cc:686] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 LEADER]: Becoming Leader. State: Replica: 02ea3b8634c246e6b6f66d5c3ba8322a, State: Running, Role: LEADER
I20260928 02:44:20.391598 18434 consensus_queue.cc:237] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:20.394838 17951 catalog_manager.cc:5495] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a reported cstate change: term changed from 0 to 1, leader changed from <none> to 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3). New cstate: current_term: 1 leader_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } health_report { overall_health: UNKNOWN } } }
I20260928 02:44:20.411361 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.407841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.2.123.3:35451)
0928 02:44:20.407963 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:20.407967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:20.407980 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:44:20.408060 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:20.408062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:20.408116 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:20.408198 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:20.408441 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.408963 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.409827 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:20.410147 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:20.410242 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:20.410510 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:20.410974 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:20.411069 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:20.411151 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:44:20.411202 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:44:20.417227 18230 raft_consensus.cc:1261] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 1 FOLLOWER]: Refusing update from remote peer 02ea3b8634c246e6b6f66d5c3ba8322a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:20.417227 18096 raft_consensus.cc:1261] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Refusing update from remote peer 02ea3b8634c246e6b6f66d5c3ba8322a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:20.417901 18434 consensus_queue.cc:1035] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [LEADER]: Connected to new peer: Peer: permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:20.418138 18437 consensus_queue.cc:1035] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:20.425846 18442 mvcc.cc:205] Tried to move back new op lower bound from 7334147933861359616 to 7334147933766643712. Current Snapshot: MvccSnapshot[applied={T|T < 7334147933861359616}]
I20260928 02:44:20.437038 17951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:20.437254 17951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:20.440127 17951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Sending DeleteTablet for 3 replicas of tablet 5ceb51e6fdd545a5be8bce4d87c274b2
I20260928 02:44:20.440960 18344 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ceb51e6fdd545a5be8bce4d87c274b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:20 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:39072
I20260928 02:44:20.440991 18076 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ceb51e6fdd545a5be8bce4d87c274b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:20 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:57924
I20260928 02:44:20.441430 18449 tablet_replica.cc:330] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: stopping tablet replica
I20260928 02:44:20.441458 18210 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ceb51e6fdd545a5be8bce4d87c274b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:20 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:46806
I20260928 02:44:20.441747 18449 raft_consensus.cc:2229] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:20.441910 18451 tablet_replica.cc:330] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: stopping tablet replica
I20260928 02:44:20.442217 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 18013
I20260928 02:44:20.442217 18449 raft_consensus.cc:2258] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:20.442268 18451 raft_consensus.cc:2229] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:20.442790 18451 raft_consensus.cc:2258] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:20.444103 18449 ts_tablet_manager.cc:1905] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:20.444170 18451 ts_tablet_manager.cc:1905] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:20.446915 18451 ts_tablet_manager.cc:1918] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:20.446918 18449 ts_tablet_manager.cc:1918] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:20.447021 18449 log.cc:1196] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/5ceb51e6fdd545a5be8bce4d87c274b2
I20260928 02:44:20.447065 18451 log.cc:1196] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/5ceb51e6fdd545a5be8bce4d87c274b2
I20260928 02:44:20.447410 18451 ts_tablet_manager.cc:1939] T 5ceb51e6fdd545a5be8bce4d87c274b2 P e1a3180e712a4349841514cf01272c26: Deleting consensus metadata
I20260928 02:44:20.447448 18449 ts_tablet_manager.cc:1939] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 02ea3b8634c246e6b6f66d5c3ba8322a: Deleting consensus metadata
I20260928 02:44:20.448794 17935 catalog_manager.cc:4841] TS 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451): tablet 5ceb51e6fdd545a5be8bce4d87c274b2 (table test-table [id=bf55a669e0374b26bad425e0674a25c4]) successfully deleted
I20260928 02:44:20.449023 17936 catalog_manager.cc:4841] TS e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729): tablet 5ceb51e6fdd545a5be8bce4d87c274b2 (table test-table [id=bf55a669e0374b26bad425e0674a25c4]) successfully deleted
I20260928 02:44:20.450577 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 18147
W20260928 02:44:20.450623 17936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:44653 (error 108)
I20260928 02:44:20.455984 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 18281
I20260928 02:44:20.456073 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.455730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:44653 (local address 127.0.0.1:57928)
0928 02:44:20.455886 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:20.455965 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:44:20.456647 17936 catalog_manager.cc:4584] TS 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ceb51e6fdd545a5be8bce4d87c274b2: Network error: Client connection negotiation failed: client connection to 127.2.123.1:44653: connect: Connection refused (error 111)
I20260928 02:44:20.461028 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 17919
I20260928 02:44:40.466292 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41415
--webserver_interface=127.2.123.62
--webserver_port=38953
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:41415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:40.574298 18453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:40.574571 18453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:40.574626 18453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:40.578642 18453 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:44:40.578708 18453 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:44:40.578730 18453 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:44:40.578747 18453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:40.578779 18453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:40.578812 18453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:40.578841 18453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:40.578863 18453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:40.584179 18453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:41415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=38953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:40.585287 18453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:40.586326 18453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:44:40.598989 18453 server_base.cc:1034] running on GCE node
W20260928 02:44:40.599009 18462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:40.598997 18459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:40.598989 18460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:40.599889 18453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:40.601003 18453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:40.602154 18453 hybrid_clock.cc:648] HybridClock initialized: now 1790563480602134 us; error 32 us; skew 500 ppm
Sep 28 02:44:40 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:40.605085 18453 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:44:40.606117 18453 webserver.cc:460] Webserver started at http://127.2.123.62:38953/ using document root <none> and password file <none>
I20260928 02:44:40.606680 18453 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:40.606770 18453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:40.610809 18453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:44:40.612938 18469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:40.614216 18453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:44:40.614331 18453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "38c3448b1a0944a389bf7df4419b2feb"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "8ae8877a42dec5eaab740c5e5a8e30bd"
server_key_iv: "2ef0decc112e008581306e34bf35f57b"
server_key_version: "encryptionkey@0"
I20260928 02:44:40.614915 18453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:40.639791 18453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:40.642899 18453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:40.643085 18453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:40.650188 18453 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:41415
I20260928 02:44:40.650454 18521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:41415 every 8 connection(s)
I20260928 02:44:40.651300 18453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:40.652935 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 18453
I20260928 02:44:40.653482 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:44653
--local_ip_for_outbound_sockets=127.2.123.1
--tserver_master_addrs=127.2.123.62:41415
--webserver_port=37275
--webserver_interface=127.2.123.1
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:40.656518 18522 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:44:40.658629 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Bootstrap starting.
I20260928 02:44:40.676079 18522 log.cc:824] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:40.684782 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:40.685145 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Bootstrap complete.
I20260928 02:44:40.688995 18522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:44:40.689380 18522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c3448b1a0944a389bf7df4419b2feb, State: Initialized, Role: FOLLOWER
I20260928 02:44:40.689851 18522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:44:40.689994 18522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:40.690076 18522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:40.690171 18522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:44:40.691939 18522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:44:40.692247 18522 leader_election.cc:302] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38c3448b1a0944a389bf7df4419b2feb; no voters:
I20260928 02:44:40.692518 18522 leader_election.cc:288] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:44:40.692617 18526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:44:40.692812 18526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 2 LEADER]: Becoming Leader. State: Replica: 38c3448b1a0944a389bf7df4419b2feb, State: Running, Role: LEADER
I20260928 02:44:40.693158 18526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:44:40.693763 18522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:40.694720 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38c3448b1a0944a389bf7df4419b2feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } } }
I20260928 02:44:40.694829 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:40.695062 18528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38c3448b1a0944a389bf7df4419b2feb. Latest consensus state: current_term: 2 leader_uuid: "38c3448b1a0944a389bf7df4419b2feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } } }
I20260928 02:44:40.695138 18528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:40.695726 18535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:40.699558 18535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:40.700031 18535 catalog_manager.cc:1259] Loaded cluster ID: 9606a57387234635ac01da8685b8a321
I20260928 02:44:40.700079 18535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:44:40.701785 18535 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:40.702401 18535 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Loaded TSK: 0
I20260928 02:44:40.721479 18535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Generated new TSK 1
I20260928 02:44:40.721740 18535 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:44:40.781076 18524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:40.781324 18524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:40.781366 18524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:40.785379 18524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:40.785444 18524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:40.785502 18524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:44:40.790688 18524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:44653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=37275
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:40.791765 18524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:40.792775 18524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:40.803628 18549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:40.803658 18552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:40.803678 18550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:40.804396 18524 server_base.cc:1034] running on GCE node
I20260928 02:44:40.804790 18524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:40.805441 18524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:40.806612 18524 hybrid_clock.cc:648] HybridClock initialized: now 1790563480806588 us; error 39 us; skew 500 ppm
Sep 28 02:44:40 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:40.809310 18524 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:44:40.810279 18524 webserver.cc:460] Webserver started at http://127.2.123.1:37275/ using document root <none> and password file <none>
I20260928 02:44:40.810809 18524 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:40.810880 18524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:40.814672 18524 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:44:40.816816 18559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:40.818208 18524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:44:40.818298 18524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "758117e6d8c3431280e882c8a28ebc2e"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "11d3be75389bfaa3df7f2d5c3367afb3"
server_key_iv: "385a79305f13a8fa9eecf0aeb775bec8"
server_key_version: "encryptionkey@0"
I20260928 02:44:40.818859 18524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:40.842936 18524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:40.846097 18524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:40.846274 18524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:40.846902 18524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:40.848037 18566 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:44:40.852246 18524 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:44:40.852314 18524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260928 02:44:40.852365 18524 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:44:40.854107 18524 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:44:40.854171 18524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260928 02:44:40.854432 18566 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Bootstrap starting.
I20260928 02:44:40.862850 18524 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:44653
I20260928 02:44:40.862895 18673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:44653 every 8 connection(s)
I20260928 02:44:40.863826 18524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:40.870415 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 18524
I20260928 02:44:40.870914 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:34729
--local_ip_for_outbound_sockets=127.2.123.2
--tserver_master_addrs=127.2.123.62:41415
--webserver_port=42799
--webserver_interface=127.2.123.2
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:40.873895 18566 log.cc:824] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:44:40 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:40.881353 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:40.865998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:39653)
0928 02:44:40.866956 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:40.866994 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:40.867757 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:40.868787 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:40.868795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:40.869184 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:40.869634 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:40.869647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.871125 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:40.871128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:40.872003 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:40.872012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.872975 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:40.872979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:40.873171 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:40.873818 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:40.873839 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:40.875454 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:40.877939 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:40.877944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:40.877957 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:40.878298 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:40.878685 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:40.878688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:40.878690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:40.878860 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:40.879597 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:40.879603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:40.879927 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 02:44:40.880119 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 02:44:40.880439 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":117,"threads_started":1}
I20260928 02:44:40.881510 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:40.866024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:39653 (local address 127.2.123.62:41415)
0928 02:44:40.866408 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:40.866427 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:40.867105 (+ 678us) server_negotiation.cc:407] Connection header received
0928 02:44:40.868136 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:40.868146 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:40.868426 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:40.868711 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:40.869790 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.870854 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:40.872115 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.872817 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:40.872973 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:40.875667 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:40.875675 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:40.875693 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:40.875727 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:40.877813 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:40.878418 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:40.878424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:40.878430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:40.878589 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:40.878967 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:40.878971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:40.878972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:40.879471 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:40.879609 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:40.880303 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 02:44:40.880630 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":130,"threads_started":1}
I20260928 02:44:40.881991 18566 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:40.882721 18566 tablet_bootstrap.cc:492] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Bootstrap complete.
I20260928 02:44:40.883761 18566 ts_tablet_manager.cc:1397] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.012s
I20260928 02:44:40.884522 18674 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41415
I20260928 02:44:40.884763 18674 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:40.885278 18674 heartbeater.cc:511] Master 127.2.123.62:41415 requested a full tablet report, sending...
I20260928 02:44:40.886222 18566 raft_consensus.cc:348] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:40.886657 18566 raft_consensus.cc:729] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758117e6d8c3431280e882c8a28ebc2e, State: Initialized, Role: FOLLOWER
I20260928 02:44:40.887149 18566 consensus_queue.cc:260] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:40.887903 18486 ts_manager.cc:194] Registered new tserver with Master: 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653)
I20260928 02:44:40.888000 18566 ts_tablet_manager.cc:1428] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 02:44:40.891716 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:39653
I20260928 02:44:40.895051 18674 heartbeater.cc:503] Master 127.2.123.62:41415 was elected leader, sending a full tablet report...
Sep 28 02:44:40 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.1@KRBTEST.COM
I20260928 02:44:40.901422 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:40.891045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:44653 (local address 127.0.0.1:46722)
0928 02:44:40.891406 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:40.891447 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:40.891564 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:40.892157 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:40.892162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:40.892537 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:40.892651 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:40.892657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.893902 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:40.893905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:40.894931 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:40.894942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.895651 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:40.895654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:40.895762 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:40.896315 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:40.896329 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:40.897932 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:40.899846 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:40.899853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:40.899857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:40.900069 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:40.900367 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:40.900370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:40.900371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:40.900449 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:40.900906 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:40.900911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:40.901009 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 02:44:40.901114 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:44:40.901294 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260928 02:44:40.901526 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:40.891120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.2.123.1:44653)
0928 02:44:40.891763 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:40.891782 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:40.891798 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:44:40.891858 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:40.891862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:40.892017 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:40.892134 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:40.892758 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.893780 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:40.895076 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.895556 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:40.895670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:40.898114 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:40.898124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:40.898135 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:40.898160 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:40.899744 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:40.900178 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:40.900184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:40.900185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:40.900276 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:40.900524 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:40.900526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:40.900528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:40.900796 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:40.900927 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:40.901206 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:44:40.901367 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":564,"thread_start_us":100,"threads_started":1}
I20260928 02:44:40.902362 18594 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ceb51e6fdd545a5be8bce4d87c274b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:20 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:46722
I20260928 02:44:40.903163 18683 tablet_replica.cc:330] stopping tablet replica
I20260928 02:44:40.903431 18683 raft_consensus.cc:2229] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:40.903650 18683 raft_consensus.cc:2258] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:40.904614 18683 ts_tablet_manager.cc:1905] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:40.908118 18683 ts_tablet_manager.cc:1918] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:40.908207 18683 log.cc:1196] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal/wals/5ceb51e6fdd545a5be8bce4d87c274b2
I20260928 02:44:40.908429 18683 ts_tablet_manager.cc:1939] T 5ceb51e6fdd545a5be8bce4d87c274b2 P 758117e6d8c3431280e882c8a28ebc2e: Deleting consensus metadata
I20260928 02:44:40.910113 18471 catalog_manager.cc:4841] TS 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653): tablet 5ceb51e6fdd545a5be8bce4d87c274b2 (table test-table [id=bf55a669e0374b26bad425e0674a25c4]) successfully deleted
W20260928 02:44:40.987856 18678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:40.988145 18678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:40.988204 18678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:40.992390 18678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:40.992465 18678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:40.992558 18678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:44:40.998222 18678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:34729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=42799
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:40.999294 18678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:41.000283 18678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:41.009617 18689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:41.009603 18691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:41.009603 18688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:41.009634 18678 server_base.cc:1034] running on GCE node
I20260928 02:44:41.010277 18678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:41.010952 18678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:41.012166 18678 hybrid_clock.cc:648] HybridClock initialized: now 1790563481012121 us; error 56 us; skew 500 ppm
Sep 28 02:44:41 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:41.014883 18678 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:44:41.015784 18678 webserver.cc:460] Webserver started at http://127.2.123.2:42799/ using document root <none> and password file <none>
I20260928 02:44:41.016340 18678 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:41.016429 18678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:41.020511 18678 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:44:41.022680 18698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.024031 18678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:44:41.024204 18678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "e1a3180e712a4349841514cf01272c26"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "f568725d75fd9b60264351c9fb428276"
server_key_iv: "3b4e0cfb9ac1a60b0ef26394e187598d"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.024745 18678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:41.056725 18678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:41.059978 18678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:41.060168 18678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:41.060763 18678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:41.061605 18678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:41.061652 18678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.061704 18678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:41.061733 18678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.070133 18678 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:34729
I20260928 02:44:41.070173 18811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:34729 every 8 connection(s)
I20260928 02:44:41.071166 18678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:41.078227 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 18678
I20260928 02:44:41.078626 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:35451
--local_ip_for_outbound_sockets=127.2.123.3
--tserver_master_addrs=127.2.123.62:41415
--webserver_port=45847
--webserver_interface=127.2.123.3
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:44:41 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:41.085873 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.072946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:45553 (local address 127.2.123.62:41415)
0928 02:44:41.073097 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.073102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.073986 (+ 884us) server_negotiation.cc:407] Connection header received
0928 02:44:41.075319 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.075322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.075374 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.075455 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:41.076768 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.077315 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.078637 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.078969 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.079082 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.081772 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.081776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.081779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.081812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.083355 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.083997 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.084000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.084002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.084086 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.084700 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.084703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.084704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.084904 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.084991 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.085628 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.085751 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:44:41.087212 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.073357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:45553)
0928 02:44:41.073808 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.073853 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.075101 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.075673 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.075688 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.076066 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:41.076607 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.076622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.077434 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.077437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.078515 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.078524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.079124 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.079127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.079336 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.079908 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:41.079925 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:41.081574 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:41.083475 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.083481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.083500 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.083879 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.084186 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.084190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.084200 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.084539 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.085011 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:41.085018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:41.085369 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.085595 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.086032 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260928 02:44:41.088404 18812 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41415
I20260928 02:44:41.088629 18812 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:41.089099 18812 heartbeater.cc:511] Master 127.2.123.62:41415 requested a full tablet report, sending...
I20260928 02:44:41.090350 18486 ts_manager.cc:194] Registered new tserver with Master: e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729)
I20260928 02:44:41.091094 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:45553
W20260928 02:44:41.190515 18815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:41.190802 18815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:41.190845 18815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:41.194929 18815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:41.194994 18815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:41.195060 18815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:44:41.200448 18815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=45847
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:44:41.201510 18815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:41.202538 18815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:41.212278 18820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:41.212337 18821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:41.212280 18823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:41.212317 18815 server_base.cc:1034] running on GCE node
I20260928 02:44:41.212946 18815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:41.213608 18815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:41.214787 18815 hybrid_clock.cc:648] HybridClock initialized: now 1790563481214749 us; error 44 us; skew 500 ppm
Sep 28 02:44:41 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:41.217610 18815 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:44:41.218631 18815 webserver.cc:460] Webserver started at http://127.2.123.3:45847/ using document root <none> and password file <none>
I20260928 02:44:41.219295 18815 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:41.219398 18815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:41.223542 18815 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:44:41.225592 18830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.226979 18815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:44:41.227097 18815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "02ea3b8634c246e6b6f66d5c3ba8322a"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "a7217c00aedaaa51b9d9a90b76f2c21e"
server_key_iv: "472c58ab463f6f58a6b8b9ad65f35cdd"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.227769 18815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:41.239900 18815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:41.242910 18815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:41.243091 18815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:41.243667 18815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:41.244464 18815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:41.244510 18815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.244561 18815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:41.244590 18815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.253163 18815 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:35451
I20260928 02:44:41.253216 18943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:35451 every 8 connection(s)
I20260928 02:44:41.254206 18815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:41.255221 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 18815
I20260928 02:44:41.264724 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.255952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.2.123.62:41415)
0928 02:44:41.256097 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.256102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.256295 (+ 193us) server_negotiation.cc:407] Connection header received
0928 02:44:41.256435 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.256438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.256486 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.256557 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:41.257497 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.258350 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.259224 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.259563 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.259669 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.261348 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.261355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.261358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.261387 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.263076 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.263551 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.263555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.263557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.263652 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.263907 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.263910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.263912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.264090 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.264364 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.264443 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.264560 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 02:44:41 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:44:41.269307 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.256190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:59203 (local address 127.2.123.62:41415)
0928 02:44:41.256406 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.256410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.257075 (+ 665us) server_negotiation.cc:407] Connection header received
0928 02:44:41.258671 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.258674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.258727 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.258806 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:41.260288 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.261097 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.262966 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.263329 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.263441 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.266269 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.266271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.266273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.266295 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.267543 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.267989 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.267991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.267992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.268055 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.268345 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.268347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.268348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.268511 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.268582 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.269037 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.269138 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260928 02:44:41.270089 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.256474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:59203)
0928 02:44:41.256941 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.256981 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.258477 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.258979 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.258986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.259490 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:41.260110 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.260131 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.261275 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.261280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.262820 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.262832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.263478 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.263481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.263675 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.264292 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:41.264311 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:41.266084 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:41.267647 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.267652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.267665 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.267908 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.268117 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.268119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.268120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.268261 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.268581 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:41.268585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:41.268818 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.268964 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.269241 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":142,"threads_started":1}
I20260928 02:44:41.271153 18944 heartbeater.cc:347] Connected to a master server at 127.2.123.62:41415
I20260928 02:44:41.271384 18944 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:41.271860 18944 heartbeater.cc:511] Master 127.2.123.62:41415 requested a full tablet report, sending...
I20260928 02:44:41.272806 18486 ts_manager.cc:194] Registered new tserver with Master: 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451)
I20260928 02:44:41.273336 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:59203
I20260928 02:44:41.278442 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:44:41.282763 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.279028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.2.123.62:41415)
0928 02:44:41.279155 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.279159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.279378 (+ 219us) server_negotiation.cc:407] Connection header received
0928 02:44:41.279514 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.279517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.279641 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.279728 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:41.279976 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.280492 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.281511 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.281768 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.281859 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.282293 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:41.282425 (+ 132us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:44:41.282636 (+ 211us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34932: token expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 02:44:41.282829 18947 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34932: token expired
W20260928 02:44:41.283197 18413 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.123.62:41415)
I20260928 02:44:41.289945 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.283531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.2.123.62:41415)
0928 02:44:41.283633 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.283637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.283648 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:44:41.283725 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.283727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.283762 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.283822 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:41.284401 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.284922 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.285689 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.285892 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.286010 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.287244 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.287247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.287249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.287272 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.288538 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.288927 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.288929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.288931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.289002 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.289276 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.289278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.289279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.289441 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.289537 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.289730 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.289825 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:44:41.292361 18486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:44:41.295780 18486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:41.303731 18594 tablet_service.cc:1467] Processing CreateTablet for tablet b7366a13cf8f46c5926e0cf3c98a3325 (DEFAULT_TABLE table=test-table [id=af81c8fabf614134ae14af07e0b11889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:41.304407 18594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7366a13cf8f46c5926e0cf3c98a3325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:41.307288 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.302357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35451 (local address 127.0.0.1:56906)
0928 02:44:41.302468 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.302483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.302558 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.303383 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.303386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.303406 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.303514 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.303521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.304645 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.304649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.305709 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.305719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.306845 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.306849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.306943 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.306984 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.307069 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.307162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:44:41.307284 18566 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Bootstrap starting.
I20260928 02:44:41.307494 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.302776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:34729 (local address 127.0.0.1:59598)
0928 02:44:41.303000 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.303014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.303107 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.303799 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.303802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.303820 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.303906 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.303912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.305100 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.305104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.306139 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.306148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.307088 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.307092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.307183 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.307205 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.307272 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.307353 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260928 02:44:41.307902 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.302926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.2.123.2:34729)
0928 02:44:41.303370 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.303381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.303403 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:44:41.303475 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.303480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.303662 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.303813 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.303985 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.304985 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.306252 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.306797 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.306899 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.307039 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.307659 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.307772 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":110,"threads_started":1}
I20260928 02:44:41.308074 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.302513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.2.123.3:35451)
0928 02:44:41.302883 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.302896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.302921 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:44:41.302999 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.303005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.303231 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.303394 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.303617 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.304538 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.305869 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.306721 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.306833 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.306995 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.307842 (+ 847us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.307942 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":146,"threads_started":1}
I20260928 02:44:41.308171 18566 tablet_bootstrap.cc:654] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:41.309738 18566 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: No bootstrap required, opened a new log
I20260928 02:44:41.309827 18566 ts_tablet_manager.cc:1397] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:44:41.310293 18566 raft_consensus.cc:348] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.310423 18566 raft_consensus.cc:374] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:41.310460 18566 raft_consensus.cc:729] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758117e6d8c3431280e882c8a28ebc2e, State: Initialized, Role: FOLLOWER
I20260928 02:44:41.310591 18566 consensus_queue.cc:260] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.310837 18566 ts_tablet_manager.cc:1428] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:44:41.310915 18878 tablet_service.cc:1467] Processing CreateTablet for tablet b7366a13cf8f46c5926e0cf3c98a3325 (DEFAULT_TABLE table=test-table [id=af81c8fabf614134ae14af07e0b11889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:41.310915 18746 tablet_service.cc:1467] Processing CreateTablet for tablet b7366a13cf8f46c5926e0cf3c98a3325 (DEFAULT_TABLE table=test-table [id=af81c8fabf614134ae14af07e0b11889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:41.311959 18746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7366a13cf8f46c5926e0cf3c98a3325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:41.311978 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7366a13cf8f46c5926e0cf3c98a3325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:41.318805 18957 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: Bootstrap starting.
I20260928 02:44:41.318879 18958 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: Bootstrap starting.
I20260928 02:44:41.321185 18957 tablet_bootstrap.cc:654] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:41.321236 18958 tablet_bootstrap.cc:654] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:41.322126 18957 log.cc:824] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:41.322140 18958 log.cc:824] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:41.323931 18958 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: No bootstrap required, opened a new log
I20260928 02:44:41.324013 18957 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: No bootstrap required, opened a new log
I20260928 02:44:41.324118 18958 ts_tablet_manager.cc:1397] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 02:44:41.324180 18957 ts_tablet_manager.cc:1397] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 02:44:41.327131 18958 raft_consensus.cc:348] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.327131 18957 raft_consensus.cc:348] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.327394 18958 raft_consensus.cc:374] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:41.327399 18957 raft_consensus.cc:374] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:41.327450 18957 raft_consensus.cc:729] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a3180e712a4349841514cf01272c26, State: Initialized, Role: FOLLOWER
I20260928 02:44:41.327487 18958 raft_consensus.cc:729] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ea3b8634c246e6b6f66d5c3ba8322a, State: Initialized, Role: FOLLOWER
I20260928 02:44:41.327913 18957 consensus_queue.cc:260] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.327912 18958 consensus_queue.cc:260] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.328558 18944 heartbeater.cc:503] Master 127.2.123.62:41415 was elected leader, sending a full tablet report...
I20260928 02:44:41.328750 18812 heartbeater.cc:503] Master 127.2.123.62:41415 was elected leader, sending a full tablet report...
I20260928 02:44:41.328863 18958 ts_tablet_manager.cc:1428] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260928 02:44:41.328864 18957 ts_tablet_manager.cc:1428] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260928 02:44:41.347012 18962 raft_consensus.cc:482] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:41.347213 18962 raft_consensus.cc:504] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.348338 18962 leader_election.cc:288] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451), 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653)
I20260928 02:44:41.352700 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.348566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35451 (local address 127.2.123.2:44477)
0928 02:44:41.348682 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.348698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.348797 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.349050 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.349058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.349078 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.349200 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.349207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.350179 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.350189 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.351421 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.351431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.352273 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.352276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.352386 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.352410 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.352480 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.352555 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":24}
I20260928 02:44:41.352766 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.348699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:44477 (local address 127.2.123.3:35451)
0928 02:44:41.348818 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.348823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.348841 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:44:41.348897 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.348900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.348947 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.349026 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.349326 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.350051 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.351552 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.352147 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.352258 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.352323 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.352573 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.352647 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:44:41.353096 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.348664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:54329 (local address 127.2.123.1:44653)
0928 02:44:41.348786 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.348790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.349155 (+ 365us) server_negotiation.cc:407] Connection header received
0928 02:44:41.349236 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.349238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.349300 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.349400 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.349655 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.350356 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.351731 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.352379 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.352467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.352641 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.352903 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.352981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:44:41.353431 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7366a13cf8f46c5926e0cf3c98a3325" candidate_uuid: "e1a3180e712a4349841514cf01272c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" is_pre_election: true
I20260928 02:44:41.353524 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.348566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:44653 (local address 127.2.123.2:54329)
0928 02:44:41.349062 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.349073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.349156 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.349441 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.349443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.349461 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.349552 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.349557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.350464 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.350467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.351605 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.351612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.352488 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.352491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.352751 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.352773 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.353334 (+ 561us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.353400 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":174,"thread_start_us":138,"threads_started":1}
I20260928 02:44:41.353765 18898 raft_consensus.cc:2454] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a3180e712a4349841514cf01272c26 in term 0.
I20260928 02:44:41.354058 18614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7366a13cf8f46c5926e0cf3c98a3325" candidate_uuid: "e1a3180e712a4349841514cf01272c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758117e6d8c3431280e882c8a28ebc2e" is_pre_election: true
I20260928 02:44:41.354264 18699 leader_election.cc:302] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ea3b8634c246e6b6f66d5c3ba8322a, e1a3180e712a4349841514cf01272c26; no voters:
I20260928 02:44:41.354346 18614 raft_consensus.cc:2454] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a3180e712a4349841514cf01272c26 in term 0.
I20260928 02:44:41.354544 18962 raft_consensus.cc:2781] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:41.354655 18962 raft_consensus.cc:482] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:41.354715 18962 raft_consensus.cc:3037] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:41.355739 18962 raft_consensus.cc:504] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.356118 18962 leader_election.cc:288] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [CANDIDATE]: Term 1 election: Requested vote from peers 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451), 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653)
I20260928 02:44:41.356422 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7366a13cf8f46c5926e0cf3c98a3325" candidate_uuid: "e1a3180e712a4349841514cf01272c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a"
I20260928 02:44:41.356443 18614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7366a13cf8f46c5926e0cf3c98a3325" candidate_uuid: "e1a3180e712a4349841514cf01272c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758117e6d8c3431280e882c8a28ebc2e"
I20260928 02:44:41.356542 18614 raft_consensus.cc:3037] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:41.356539 18898 raft_consensus.cc:3037] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:41.357502 18898 raft_consensus.cc:2454] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a3180e712a4349841514cf01272c26 in term 1.
I20260928 02:44:41.357568 18614 raft_consensus.cc:2454] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a3180e712a4349841514cf01272c26 in term 1.
I20260928 02:44:41.357779 18699 leader_election.cc:302] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ea3b8634c246e6b6f66d5c3ba8322a, e1a3180e712a4349841514cf01272c26; no voters:
I20260928 02:44:41.358012 18962 raft_consensus.cc:2781] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:41.358214 18962 raft_consensus.cc:686] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 1 LEADER]: Becoming Leader. State: Replica: e1a3180e712a4349841514cf01272c26, State: Running, Role: LEADER
I20260928 02:44:41.358517 18962 consensus_queue.cc:237] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:44:41.361544 18486 catalog_manager.cc:5495] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a3180e712a4349841514cf01272c26 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "e1a3180e712a4349841514cf01272c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } health_report { overall_health: UNKNOWN } } }
W20260928 02:44:41.364952 18675 tablet.cc:2369] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:44:41.373523 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.370096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.2.123.2:34729)
0928 02:44:41.370242 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.370246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.370273 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:44:41.370337 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.370340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.370388 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.370473 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:41.370658 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.371179 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.372070 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.372318 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.372399 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.372724 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:41.373117 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:41.373261 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.373344 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.373394 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:44:41.379596 18898 raft_consensus.cc:1261] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 FOLLOWER]: Refusing update from remote peer e1a3180e712a4349841514cf01272c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:41.379585 18614 raft_consensus.cc:1261] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Refusing update from remote peer e1a3180e712a4349841514cf01272c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:41.380430 18964 consensus_queue.cc:1035] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:41.380682 18962 consensus_queue.cc:1035] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:41.387244 18968 mvcc.cc:205] Tried to move back new op lower bound from 7334148019722170368 to 7334148019646787584. Current Snapshot: MvccSnapshot[applied={T|T < 7334148019722170368}]
I20260928 02:44:41.388230 18967 mvcc.cc:205] Tried to move back new op lower bound from 7334148019722170368 to 7334148019646787584. Current Snapshot: MvccSnapshot[applied={T|T < 7334148019722170368 or (T in {7334148019722170368})}]
I20260928 02:44:41.395629 18486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:41.395802 18486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:41.398054 18486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Sending DeleteTablet for 3 replicas of tablet b7366a13cf8f46c5926e0cf3c98a3325
I20260928 02:44:41.398862 18746 tablet_service.cc:1514] Processing DeleteTablet for tablet b7366a13cf8f46c5926e0cf3c98a3325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:41 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:59598
I20260928 02:44:41.398886 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet b7366a13cf8f46c5926e0cf3c98a3325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:41 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:56906
I20260928 02:44:41.399345 18594 tablet_service.cc:1514] Processing DeleteTablet for tablet b7366a13cf8f46c5926e0cf3c98a3325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:41 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:46722
I20260928 02:44:41.399385 18978 tablet_replica.cc:330] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: stopping tablet replica
I20260928 02:44:41.399528 18683 tablet_replica.cc:330] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: stopping tablet replica
I20260928 02:44:41.399647 18683 raft_consensus.cc:2229] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:41.399650 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 18524
I20260928 02:44:41.399789 18683 raft_consensus.cc:2258] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:41.400241 18978 raft_consensus.cc:2229] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:41.400595 18978 raft_consensus.cc:2258] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:41.400938 18977 tablet_replica.cc:330] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: stopping tablet replica
I20260928 02:44:41.401304 18977 raft_consensus.cc:2229] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:41.401669 18977 raft_consensus.cc:2258] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:41.402276 18978 ts_tablet_manager.cc:1905] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:41.403148 18977 ts_tablet_manager.cc:1905] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:41.405591 18977 ts_tablet_manager.cc:1918] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:41.405685 18977 log.cc:1196] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/b7366a13cf8f46c5926e0cf3c98a3325
I20260928 02:44:41.405993 18977 ts_tablet_manager.cc:1939] T b7366a13cf8f46c5926e0cf3c98a3325 P 02ea3b8634c246e6b6f66d5c3ba8322a: Deleting consensus metadata
I20260928 02:44:41.407011 18978 ts_tablet_manager.cc:1918] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:41.407141 18978 log.cc:1196] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/b7366a13cf8f46c5926e0cf3c98a3325
I20260928 02:44:41.407186 18470 catalog_manager.cc:4841] TS 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451): tablet b7366a13cf8f46c5926e0cf3c98a3325 (table test-table [id=af81c8fabf614134ae14af07e0b11889]) successfully deleted
I20260928 02:44:41.407452 18978 ts_tablet_manager.cc:1939] T b7366a13cf8f46c5926e0cf3c98a3325 P e1a3180e712a4349841514cf01272c26: Deleting consensus metadata
W20260928 02:44:41.407589 18471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:44653 (error 108)
I20260928 02:44:41.408120 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 18678
I20260928 02:44:41.409322 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.409008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:44653 (local address 127.0.0.1:46728)
0928 02:44:41.409156 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.409236 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:44:41.409476 18471 catalog_manager.cc:4584] TS 758117e6d8c3431280e882c8a28ebc2e (127.2.123.1:44653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7366a13cf8f46c5926e0cf3c98a3325: Network error: Client connection negotiation failed: client connection to 127.2.123.1:44653: connect: Connection refused (error 111)
I20260928 02:44:41.413187 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 18815
I20260928 02:44:41.414947 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.414650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:34729 (local address 127.0.0.1:59626)
0928 02:44:41.414784 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.414861 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.2:34729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:44:41.417939 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 18453
I20260928 02:45:01.423523 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41415
--webserver_interface=127.2.123.62
--webserver_port=38953
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:41415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:01.538021 18980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:01.538317 18980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:01.538365 18980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:01.542483 18980 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:45:01.542546 18980 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:45:01.542562 18980 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:45:01.542574 18980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:01.542593 18980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:01.542610 18980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:01.542629 18980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:01.542657 18980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:01.548143 18980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:41415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=38953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:01.549281 18980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:01.550495 18980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:01.559350 18985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:01.559593 18980 server_base.cc:1034] running on GCE node
W20260928 02:45:01.559403 18986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:01.559350 18988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:01.560286 18980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:01.561478 18980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:01.562668 18980 hybrid_clock.cc:648] HybridClock initialized: now 1790563501562641 us; error 43 us; skew 500 ppm
Sep 28 02:45:01 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:01.566188 18980 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:01.567443 18980 webserver.cc:460] Webserver started at http://127.2.123.62:38953/ using document root <none> and password file <none>
I20260928 02:45:01.568239 18980 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:01.568372 18980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:01.574553 18980 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:45:01.577729 18995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:01.579620 18980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:01.579810 18980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "38c3448b1a0944a389bf7df4419b2feb"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "8ae8877a42dec5eaab740c5e5a8e30bd"
server_key_iv: "2ef0decc112e008581306e34bf35f57b"
server_key_version: "encryptionkey@0"
I20260928 02:45:01.580646 18980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:01.600404 18980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:01.603986 18980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:01.604193 18980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:01.611328 18980 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:41415
I20260928 02:45:01.611608 19047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:41415 every 8 connection(s)
I20260928 02:45:01.612438 18980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:01.617274 19048 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:45:01.619249 19048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Bootstrap starting.
I20260928 02:45:01.620647 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 18980
I20260928 02:45:01.621111 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:44653
--local_ip_for_outbound_sockets=127.2.123.1
--tserver_master_addrs=127.2.123.62:41415
--webserver_port=37275
--webserver_interface=127.2.123.1
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:45:01.637385 19048 log.cc:824] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:01.652915 19048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:45:01.653548 19048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Bootstrap complete.
I20260928 02:45:01.659299 19048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:45:01.659700 19048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c3448b1a0944a389bf7df4419b2feb, State: Initialized, Role: FOLLOWER
I20260928 02:45:01.660187 19048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:45:01.660303 19048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:01.660369 19048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:01.660459 19048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:45:01.662395 19048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:45:01.662714 19048 leader_election.cc:302] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38c3448b1a0944a389bf7df4419b2feb; no voters:
I20260928 02:45:01.663022 19048 leader_election.cc:288] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:45:01.663125 19052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:45:01.663308 19052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [term 3 LEADER]: Becoming Leader. State: Replica: 38c3448b1a0944a389bf7df4419b2feb, State: Running, Role: LEADER
I20260928 02:45:01.663678 19052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } }
I20260928 02:45:01.664304 19048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:01.664634 19052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38c3448b1a0944a389bf7df4419b2feb. Latest consensus state: current_term: 3 leader_uuid: "38c3448b1a0944a389bf7df4419b2feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } } }
I20260928 02:45:01.665328 19052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:01.665220 19053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "38c3448b1a0944a389bf7df4419b2feb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3448b1a0944a389bf7df4419b2feb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41415 } } }
I20260928 02:45:01.665913 19053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:01.666312 19061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:01.672180 19061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:01.672963 19061 catalog_manager.cc:1259] Loaded cluster ID: 9606a57387234635ac01da8685b8a321
I20260928 02:45:01.673084 19061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:01.675843 19061 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:01.676666 19061 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Loaded TSK: 0
I20260928 02:45:01.676756 19061 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38c3448b1a0944a389bf7df4419b2feb: Loaded TSK: 1
F20260928 02:45:01.677520 19061 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb5b09659ad google::LogMessage::Fail() at ??:0
@ 0x7fb5b096888a google::LogMessage::SendToLog() at ??:0
@ 0x7fb5b09654de google::LogMessage::Flush() at ??:0
@ 0x7fb5b0966a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb5b362918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb5b3621f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb5bb2ca7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb5bb2a499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb5bb2a5178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fb5bb2e6df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb5bb30582b std::function<>::operator()() at ??:0
@ 0x7fb5bb2a4b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb5bb2a5f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb5bb2a2579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb5bb2e6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb5b2bb73d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb5b2bb7c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb5b2bb93f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb5b2ba8512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb5b41ec6db start_thread at ??:0
@ 0x7fb5af46361f clone at ??:0
W20260928 02:45:01.779763 19050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:01.780184 19050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:01.780264 19050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:01.786368 19050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:01.786473 19050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:01.786577 19050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:45:01.794862 19050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:44653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=37275
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:01.796353 19050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:01.797753 19050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:01.810386 19095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:01.810510 19098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:01.810353 19096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:01.811273 19050 server_base.cc:1034] running on GCE node
I20260928 02:45:01.811724 19050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:01.812395 19050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:01.813584 19050 hybrid_clock.cc:648] HybridClock initialized: now 1790563501813552 us; error 51 us; skew 500 ppm
Sep 28 02:45:01 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:01.819654 19050 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:45:01.820657 19050 webserver.cc:460] Webserver started at http://127.2.123.1:37275/ using document root <none> and password file <none>
I20260928 02:45:01.821271 19050 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:01.821347 19050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:01.825496 19050 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:45:01.827876 19105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:01.829320 19050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:45:01.829439 19050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "758117e6d8c3431280e882c8a28ebc2e"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "11d3be75389bfaa3df7f2d5c3367afb3"
server_key_iv: "385a79305f13a8fa9eecf0aeb775bec8"
server_key_version: "encryptionkey@0"
I20260928 02:45:01.830026 19050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:01.866645 19050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:01.869881 19050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:01.870100 19050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:01.870754 19050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:01.872144 19112 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:45:01.878371 19050 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:45:01.878448 19050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 02:45:01.878506 19050 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:45:01.880252 19050 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:45:01.880304 19050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 02:45:01.880597 19112 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Bootstrap starting.
I20260928 02:45:01.889915 19050 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:44653
I20260928 02:45:01.890206 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:44653 every 8 connection(s)
I20260928 02:45:01.891158 19050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:45:01.899217 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 19050
I20260928 02:45:01.899621 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:34729
--local_ip_for_outbound_sockets=127.2.123.2
--tserver_master_addrs=127.2.123.62:41415
--webserver_port=42799
--webserver_interface=127.2.123.2
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:45:01.902441 19112 log.cc:824] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:01.911895 19112 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:45:01.912338 19112 tablet_bootstrap.cc:492] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Bootstrap complete.
I20260928 02:45:01.913650 19112 ts_tablet_manager.cc:1397] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.007s
I20260928 02:45:01.917007 19112 raft_consensus.cc:348] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:45:01.917565 19112 raft_consensus.cc:729] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758117e6d8c3431280e882c8a28ebc2e, State: Initialized, Role: FOLLOWER
I20260928 02:45:01.918161 19112 consensus_queue.cc:260] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:45:01.919246 19112 ts_tablet_manager.cc:1428] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260928 02:45:01.927800 19222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:01.893079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:40907)
0928 02:45:01.893558 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:01.893612 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:01.894492 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:01.926853 (+ 32361us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
W20260928 02:45:01.928363 19220 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:41415 (0 consecutive failures): Network error: Failed to ping master at 127.2.123.62:41415: Client connection negotiation failed: client connection to 127.2.123.62:41415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:45:01.929165 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:01.928849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:45727)
0928 02:45:01.928979 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:01.929097 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:45:01.929822 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:01.929613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:34303)
0928 02:45:01.929717 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:01.929765 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 02:45:01.930059 19220 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:45:02.013266 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:02.013554 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:02.013599 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:02.017896 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:02.017983 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:02.018069 19223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:45:02.023414 19223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:34729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=42799
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:02.024463 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:02.025472 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:02.035233 19233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:02.035251 19231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:02.035234 19230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:02.035553 19223 server_base.cc:1034] running on GCE node
I20260928 02:45:02.036007 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:02.036615 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:02.037794 19223 hybrid_clock.cc:648] HybridClock initialized: now 1790563502037777 us; error 36 us; skew 500 ppm
Sep 28 02:45:02 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563502, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:02.040488 19223 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:45:02.041368 19223 webserver.cc:460] Webserver started at http://127.2.123.2:42799/ using document root <none> and password file <none>
I20260928 02:45:02.041908 19223 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:02.041999 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:02.046015 19223 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:45:02.048107 19240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:02.049371 19223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:45:02.049482 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "e1a3180e712a4349841514cf01272c26"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "f568725d75fd9b60264351c9fb428276"
server_key_iv: "3b4e0cfb9ac1a60b0ef26394e187598d"
server_key_version: "encryptionkey@0"
I20260928 02:45:02.050071 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:02.078967 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:02.082274 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:02.082463 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:02.083088 19223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:02.083935 19223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:02.083981 19223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:02.084031 19223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:02.084060 19223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:02.092535 19223 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:34729
I20260928 02:45:02.092562 19353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:34729 every 8 connection(s)
I20260928 02:45:02.093533 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:45:02.096819 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 19223
I20260928 02:45:02.097196 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:35451
--local_ip_for_outbound_sockets=127.2.123.3
--tserver_master_addrs=127.2.123.62:41415
--webserver_port=45847
--webserver_interface=127.2.123.3
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:45:02.097543 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:02.095655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:41713)
0928 02:45:02.096114 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:02.096229 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
W20260928 02:45:02.098104 19354 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:41415 (0 consecutive failures): Network error: Failed to ping master at 127.2.123.62:41415: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
I20260928 02:45:02.098889 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:02.098600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:58139)
0928 02:45:02.098745 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:02.098806 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:45:02.099555 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:02.099326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:56647)
0928 02:45:02.099436 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:02.099491 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:45:02.099720 19354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:45:02.208951 19357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:02.209251 19357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:02.209297 19357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:02.213380 19357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:02.213454 19357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:02.213512 19357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:45:02.218909 19357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=45847
--tserver_master_addrs=127.2.123.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:02.219970 19357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:02.221002 19357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:02.230592 19365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:02.230592 19363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:02.230611 19362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:02.230875 19357 server_base.cc:1034] running on GCE node
I20260928 02:45:02.231455 19357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:02.232091 19357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:02.233245 19357 hybrid_clock.cc:648] HybridClock initialized: now 1790563502233219 us; error 46 us; skew 500 ppm
Sep 28 02:45:02 dist-test-slave-rqvb krb5kdc[17906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563502, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:02.235942 19357 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:45:02.236821 19357 webserver.cc:460] Webserver started at http://127.2.123.3:45847/ using document root <none> and password file <none>
I20260928 02:45:02.237353 19357 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:02.237419 19357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:02.241358 19357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:45:02.243454 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:02.244702 19357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 02:45:02.244832 19357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "02ea3b8634c246e6b6f66d5c3ba8322a"
format_stamp: "Formatted at 2026-09-28 02:44:19 on dist-test-slave-rqvb"
server_key: "a7217c00aedaaa51b9d9a90b76f2c21e"
server_key_iv: "472c58ab463f6f58a6b8b9ad65f35cdd"
server_key_version: "encryptionkey@0"
I20260928 02:45:02.245364 19357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:02.264299 19357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:02.267690 19357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:02.267892 19357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:02.268491 19357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:02.269316 19357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:02.269361 19357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:02.269413 19357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:02.269443 19357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:02.278617 19357 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:35451
I20260928 02:45:02.278760 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:35451 every 8 connection(s)
I20260928 02:45:02.279670 19357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:45:02.283923 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:02.282011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:37993)
0928 02:45:02.282542 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:02.282693 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
I20260928 02:45:02.284476 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 19357
W20260928 02:45:02.284662 19486 heartbeater.cc:644] Failed to heartbeat to 127.2.123.62:41415 (0 consecutive failures): Network error: Failed to ping master at 127.2.123.62:41415: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
I20260928 02:45:02.285358 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:02.285047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:41895)
0928 02:45:02.285197 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:02.285275 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 02:45:02.286022 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:02.285794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:56895)
0928 02:45:02.285891 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:02.285937 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:45:02.286260 19486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:45:02.931255 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:02.930748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:33641)
0928 02:45:02.931047 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:02.931113 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260928 02:45:03.100835 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:03.100390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:40559)
0928 02:45:03.100662 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:03.100725 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260928 02:45:03.267469 19492 raft_consensus.cc:482] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:03.267663 19492 raft_consensus.cc:504] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:45:03.268927 19492 leader_election.cc:288] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451), e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729)
Sep 28 02:45:03 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.2@KRBTEST.COM
Sep 28 02:45:03 dist-test-slave-rqvb krb5kdc[17906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.3@KRBTEST.COM
I20260928 02:45:03.284423 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:03.269185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:35451 (local address 127.2.123.1:33535)
0928 02:45:03.269692 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:03.269705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:03.269804 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:03.271507 (+ 1703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:03.271509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:03.271766 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:03.272251 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:03.272267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:03.275332 (+ 3065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:03.275336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:03.276104 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:03.276115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:03.276981 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:03.276985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:03.277066 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:03.277547 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:03.277558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:03.279153 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:03.281483 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:03.281490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:03.281505 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:03.282002 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:03.282698 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:03.282701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:03.282702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:03.282782 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:03.283472 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:03.283479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:03.283709 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 02:45:03.283873 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 02:45:03.284233 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":192,"spinlock_wait_cycles":44544,"thread_start_us":67,"threads_started":1}
I20260928 02:45:03.284478 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:03.269180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:34729 (local address 127.2.123.1:48665)
0928 02:45:03.269288 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:03.269302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:03.269410 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:03.271251 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:03.271257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:03.271767 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:03.272251 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:03.272266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:03.273639 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:03.273643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:03.275020 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:03.275042 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:03.276127 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:03.276130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:03.276314 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:03.277130 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:03.277147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:03.279017 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:03.281540 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:03.281543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:03.281545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:03.282002 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:03.282421 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:03.282424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:03.282426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:03.282573 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:03.283487 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:03.283491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:03.283709 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 02:45:03.283866 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:45:03.284317 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":155,"spinlock_wait_cycles":22784}
I20260928 02:45:03.284477 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:03.269367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:33535 (local address 127.2.123.3:35451)
0928 02:45:03.269686 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:03.269702 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:03.269803 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:45:03.270525 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:03.270531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:03.270877 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:03.271155 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:03.273672 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:03.275191 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:03.276220 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:03.276863 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:03.277017 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:03.279377 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:03.279383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:03.279398 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:03.279434 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:03.281342 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:03.282407 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:03.282412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:03.282417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:03.282569 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:03.282862 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:03.282864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:03.282866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:03.283350 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:03.283468 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:03.284027 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 02:45:03.284288 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
I20260928 02:45:03.284454 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:03.269464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:48665 (local address 127.2.123.2:34729)
0928 02:45:03.269793 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:03.269815 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:03.269848 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:45:03.270538 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:03.270543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:03.270878 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:03.271129 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:03.272428 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:03.273480 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:03.275180 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:03.275989 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:03.276173 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:03.279291 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:03.279298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:03.279317 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:03.279354 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:03.281407 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:03.282116 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:03.282123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:03.282128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:03.282272 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:03.282893 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:03.282896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:03.282897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:03.283305 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:03.283492 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:03.284021 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 02:45:03.284282 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":73,"threads_started":1}
I20260928 02:45:03.287141 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:03.286731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:60315)
0928 02:45:03.286978 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:03.287038 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
W20260928 02:45:03.287317 19106 leader_election.cc:341] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451): Not found: Tablet not found: b7366a13cf8f46c5926e0cf3c98a3325
W20260928 02:45:03.287482 19107 leader_election.cc:341] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729): Not found: Tablet not found: b7366a13cf8f46c5926e0cf3c98a3325
I20260928 02:45:03.287572 19107 leader_election.cc:302] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758117e6d8c3431280e882c8a28ebc2e; no voters: 02ea3b8634c246e6b6f66d5c3ba8322a, e1a3180e712a4349841514cf01272c26
I20260928 02:45:03.287873 19492 raft_consensus.cc:2726] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:45:03.932502 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:03.932069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:55713)
0928 02:45:03.932332 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:03.932395 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260928 02:45:04.102097 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:04.101580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:41259)
0928 02:45:04.101885 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:04.101996 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260928 02:45:04.288669 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:04.288194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:35159)
0928 02:45:04.288474 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:04.288543 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260928 02:45:04.862957 19500 raft_consensus.cc:482] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:04.863093 19500 raft_consensus.cc:504] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:45:04.863502 19500 leader_election.cc:288] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451), e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729)
W20260928 02:45:04.864413 19107 leader_election.cc:341] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729): Not found: Tablet not found: b7366a13cf8f46c5926e0cf3c98a3325
W20260928 02:45:04.864527 19106 leader_election.cc:341] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451): Not found: Tablet not found: b7366a13cf8f46c5926e0cf3c98a3325
I20260928 02:45:04.864588 19106 leader_election.cc:302] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758117e6d8c3431280e882c8a28ebc2e; no voters: 02ea3b8634c246e6b6f66d5c3ba8322a, e1a3180e712a4349841514cf01272c26
I20260928 02:45:04.864724 19500 raft_consensus.cc:2726] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:45:04.933763 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:04.933284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:33859)
0928 02:45:04.933555 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:04.933620 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260928 02:45:05.103263 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:05.102797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:55349)
0928 02:45:05.103059 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:05.103149 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260928 02:45:05.289942 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:05.289497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:60689)
0928 02:45:05.289770 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:05.289829 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260928 02:45:05.935133 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:05.934684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:34759)
0928 02:45:05.934964 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:05.935033 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260928 02:45:06.104580 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:06.104078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:59461)
0928 02:45:06.104405 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:06.104468 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
I20260928 02:45:06.291365 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:06.290849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:40321)
0928 02:45:06.291154 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:06.291224 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
I20260928 02:45:06.631193 19507 raft_consensus.cc:482] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:06.631359 19507 raft_consensus.cc:504] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:45:06.631850 19507 leader_election.cc:288] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451), e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729)
W20260928 02:45:06.632800 19106 leader_election.cc:341] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451): Not found: Tablet not found: b7366a13cf8f46c5926e0cf3c98a3325
W20260928 02:45:06.632994 19107 leader_election.cc:341] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729): Not found: Tablet not found: b7366a13cf8f46c5926e0cf3c98a3325
I20260928 02:45:06.633072 19107 leader_election.cc:302] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758117e6d8c3431280e882c8a28ebc2e; no voters: 02ea3b8634c246e6b6f66d5c3ba8322a, e1a3180e712a4349841514cf01272c26
I20260928 02:45:06.633219 19507 raft_consensus.cc:2726] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:45:06.936604 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:06.936114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:42555)
0928 02:45:06.936421 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:06.936478 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":77,"threads_started":1}
I20260928 02:45:07.105935 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:07.105478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:59721)
0928 02:45:07.105768 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:07.105834 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":127,"threads_started":1}
I20260928 02:45:07.292616 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:07.292198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:39615)
0928 02:45:07.292448 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:07.292517 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260928 02:45:07.938019 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:07.937524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:40247)
0928 02:45:07.937817 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:07.937883 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
I20260928 02:45:08.107316 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:08.106820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:43945)
0928 02:45:08.107156 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:08.107219 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":85,"threads_started":1}
I20260928 02:45:08.293815 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:08.293385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:43825)
0928 02:45:08.293655 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:08.293719 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260928 02:45:08.939270 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:08.938857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:52125)
0928 02:45:08.939113 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:08.939173 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260928 02:45:09.108603 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:09.108071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:38083)
0928 02:45:09.108432 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:09.108503 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260928 02:45:09.295207 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:09.294671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:47879)
0928 02:45:09.295029 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:09.295099 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":78,"threads_started":1}
I20260928 02:45:09.940546 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:09.940122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:51403)
0928 02:45:09.940381 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:09.940442 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260928 02:45:10.110088 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:10.109559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:57745)
0928 02:45:10.109840 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:10.109924 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260928 02:45:10.296598 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:10.296150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:56253)
0928 02:45:10.296436 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:10.296503 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260928 02:45:10.941902 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:10.941464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:60403)
0928 02:45:10.941744 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:10.941808 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260928 02:45:11.111519 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:11.111051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:53217)
0928 02:45:11.111357 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:11.111424 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":90,"threads_started":1}
I20260928 02:45:11.297906 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:11.297419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:52147)
0928 02:45:11.297742 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:11.297809 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260928 02:45:11.303023 19523 raft_consensus.cc:482] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:11.303151 19523 raft_consensus.cc:504] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ea3b8634c246e6b6f66d5c3ba8322a" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 35451 } } peers { permanent_uuid: "e1a3180e712a4349841514cf01272c26" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 34729 } } peers { permanent_uuid: "758117e6d8c3431280e882c8a28ebc2e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 44653 } }
I20260928 02:45:11.303529 19523 leader_election.cc:288] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451), e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729)
W20260928 02:45:11.304337 19106 leader_election.cc:341] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02ea3b8634c246e6b6f66d5c3ba8322a (127.2.123.3:35451): Not found: Tablet not found: b7366a13cf8f46c5926e0cf3c98a3325
W20260928 02:45:11.304440 19107 leader_election.cc:341] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a3180e712a4349841514cf01272c26 (127.2.123.2:34729): Not found: Tablet not found: b7366a13cf8f46c5926e0cf3c98a3325
I20260928 02:45:11.304548 19107 leader_election.cc:302] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 758117e6d8c3431280e882c8a28ebc2e; no voters: 02ea3b8634c246e6b6f66d5c3ba8322a, e1a3180e712a4349841514cf01272c26
I20260928 02:45:11.304731 19523 raft_consensus.cc:2726] T b7366a13cf8f46c5926e0cf3c98a3325 P 758117e6d8c3431280e882c8a28ebc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:45:11.943280 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:11.942828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:50179)
0928 02:45:11.943116 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:11.943182 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260928 02:45:12.112769 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:12.112345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:54899)
0928 02:45:12.112599 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:12.112664 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260928 02:45:12.299401 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:12.298908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:45323)
0928 02:45:12.299175 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:12.299235 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260928 02:45:12.944545 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:12.944082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:34645)
0928 02:45:12.944377 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:12.944442 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":102,"threads_started":1}
I20260928 02:45:13.113945 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:13.113502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:57609)
0928 02:45:13.113778 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:13.113835 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260928 02:45:13.300711 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:13.300245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:54431)
0928 02:45:13.300549 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:13.300614 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260928 02:45:13.945866 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:13.945432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:57699)
0928 02:45:13.945708 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:13.945769 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
I20260928 02:45:14.115121 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:14.114683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:35191)
0928 02:45:14.114957 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:14.115021 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":71,"threads_started":1}
I20260928 02:45:14.302112 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:14.301522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:56741)
0928 02:45:14.301824 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:14.301901 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":76,"threads_started":1}
I20260928 02:45:14.947148 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:14.946670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:33989)
0928 02:45:14.946925 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:14.947041 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260928 02:45:15.116326 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:15.115888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:43525)
0928 02:45:15.116160 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:15.116217 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260928 02:45:15.303308 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:15.302872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:33789)
0928 02:45:15.303156 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:15.303210 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
I20260928 02:45:15.948577 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:15.948065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:47735)
0928 02:45:15.948375 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:15.948466 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260928 02:45:16.117601 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:16.117064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:33449)
0928 02:45:16.117446 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:16.117506 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":145,"threads_started":1}
I20260928 02:45:16.304731 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:16.304276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.3:41401)
0928 02:45:16.304562 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:16.304633 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20260928 02:45:16.950155 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:16.949617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.1:43305)
0928 02:45:16.949922 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:16.950029 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":76,"threads_started":1}
I20260928 02:45:17.119009 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:17.118587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:41415 (local address 127.2.123.2:42563)
0928 02:45:17.118849 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:17.118912 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.62:41415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260928 02:45:17.292115 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 19050
I20260928 02:45:17.297135 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 19223
I20260928 02:45:17.301785 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 19357
2026-09-28T02:45:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59308 ms)
[----------] 30 tests from SecurityITest (238821 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:17 dist-test-slave-rqvb krb5kdc[19543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:17 dist-test-slave-rqvb krb5kdc[19543](info): set up 2 sockets
Sep 28 02:45:17 dist-test-slave-rqvb krb5kdc[19543](info): commencing operation
krb5kdc: starting...
W20260928 02:45:18.261945 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:18 dist-test-slave-rqvb krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:18.385421 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:46571
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:18.494498 19556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:18.494786 19556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:18.494827 19556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:18.498883 19556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:18.498955 19556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:18.498974 19556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:18.498989 19556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:18.499004 19556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:18.504330 19556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:46571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:18.505386 19556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:18.506430 19556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:18.514952 19556 server_base.cc:1034] running on GCE node
W20260928 02:45:18.515079 19562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:18.515072 19561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:18.515280 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:18.515833 19556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:18.516948 19556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:18.518110 19556 hybrid_clock.cc:648] HybridClock initialized: now 1790563518518085 us; error 39 us; skew 500 ppm
Sep 28 02:45:18 dist-test-slave-rqvb krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:18.520951 19556 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:18.521921 19556 webserver.cc:460] Webserver started at http://127.2.123.62:35517/ using document root <none> and password file <none>
I20260928 02:45:18.522552 19556 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:18.522629 19556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:18.522871 19556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:18.525022 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "62bc9c6c35034473860193684183fb52"
format_stamp: "Formatted at 2026-09-28 02:45:18 on dist-test-slave-rqvb"
server_key: "6e0e1c63ed7944b1097448339f848055"
server_key_iv: "af2841421ff903de738fe9afa2dad44e"
server_key_version: "encryptionkey@0"
I20260928 02:45:18.525619 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "62bc9c6c35034473860193684183fb52"
format_stamp: "Formatted at 2026-09-28 02:45:18 on dist-test-slave-rqvb"
server_key: "6e0e1c63ed7944b1097448339f848055"
server_key_iv: "af2841421ff903de738fe9afa2dad44e"
server_key_version: "encryptionkey@0"
I20260928 02:45:18.529572 19556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 02:45:18.532467 19571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:18.533947 19556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:45:18.534119 19556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "62bc9c6c35034473860193684183fb52"
format_stamp: "Formatted at 2026-09-28 02:45:18 on dist-test-slave-rqvb"
server_key: "6e0e1c63ed7944b1097448339f848055"
server_key_iv: "af2841421ff903de738fe9afa2dad44e"
server_key_version: "encryptionkey@0"
I20260928 02:45:18.534232 19556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:18.548388 19556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:18.551433 19556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:18.551618 19556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:18.558320 19556 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:46571
I20260928 02:45:18.558337 19623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:46571 every 8 connection(s)
I20260928 02:45:18.559316 19556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:18.561666 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 19556
I20260928 02:45:18.561787 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:18.562124 2540 external_mini_cluster.cc:1428] Setting key 44243649c7536e9b235e6219b5aeaa7f
I20260928 02:45:18.562369 19624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:18.569041 19624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52: Bootstrap starting.
Sep 28 02:45:18 dist-test-slave-rqvb krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:18.571571 19624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:18.572312 19624 log.cc:824] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:18.574314 19624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52: No bootstrap required, opened a new log
I20260928 02:45:18.576606 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:18.563573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.2.123.62:46571)
0928 02:45:18.564050 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:18.564066 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:18.564103 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:45:18.565110 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:18.565130 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:18.565486 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:18.565808 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:18.566637 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:18.567618 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:18.568468 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:18.568949 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:18.569098 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:18.571951 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:18.571957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:18.571971 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:18.571999 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:18.573750 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:18.574275 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:18.574280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:18.574285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:18.574430 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:18.574743 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:18.574746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:18.574748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:18.575132 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:18.575304 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:18.575595 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 02:45:18.575867 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":154,"threads_started":1}
I20260928 02:45:18.577862 19624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bc9c6c35034473860193684183fb52" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46571 } }
I20260928 02:45:18.578128 19624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:18.578197 19624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bc9c6c35034473860193684183fb52, State: Initialized, Role: FOLLOWER
I20260928 02:45:18.578640 19624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bc9c6c35034473860193684183fb52" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46571 } }
I20260928 02:45:18.578797 19624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:18.578859 19624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:18.578958 19624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:18.579805 19624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bc9c6c35034473860193684183fb52" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46571 } }
I20260928 02:45:18.580139 19624 leader_election.cc:302] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62bc9c6c35034473860193684183fb52; no voters:
I20260928 02:45:18.580408 19624 leader_election.cc:288] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:18.580590 19629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:18.580834 19629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [term 1 LEADER]: Becoming Leader. State: Replica: 62bc9c6c35034473860193684183fb52, State: Running, Role: LEADER
I20260928 02:45:18.581141 19629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bc9c6c35034473860193684183fb52" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46571 } }
I20260928 02:45:18.581552 19624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:18.582983 19631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62bc9c6c35034473860193684183fb52. Latest consensus state: current_term: 1 leader_uuid: "62bc9c6c35034473860193684183fb52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bc9c6c35034473860193684183fb52" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46571 } } }
I20260928 02:45:18.583029 19630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62bc9c6c35034473860193684183fb52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bc9c6c35034473860193684183fb52" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 46571 } } }
I20260928 02:45:18.583122 19630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:18.583106 19631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52 [sys.catalog]: This master's current role is: LEADER
W20260928 02:45:18.586229 19644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:18.586289 19644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:18.586352 19643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:18.587038 19643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:18.592063 19643 catalog_manager.cc:1347] Generated new cluster ID: 81d2605a87634206bfef0561730ac19d
I20260928 02:45:18.592118 19643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:18.621585 19643 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:18.622987 19643 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:18.633260 19643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52: Generated new TSK 0
I20260928 02:45:18.634200 19643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:45:18.668412 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46571
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:18.779837 19650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:18.780143 19650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:18.780191 19650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:18.784556 19650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:18.784627 19650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:18.784688 19650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:45:18.791064 19650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:18.792098 19650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:18.793075 19650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:18.802500 19656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:18.802493 19655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:18.802528 19658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:18.802794 19650 server_base.cc:1034] running on GCE node
I20260928 02:45:18.803212 19650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:18.803885 19650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:18.805085 19650 hybrid_clock.cc:648] HybridClock initialized: now 1790563518805062 us; error 39 us; skew 500 ppm
Sep 28 02:45:18 dist-test-slave-rqvb krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:18.807824 19650 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:45:18.808727 19650 webserver.cc:460] Webserver started at http://127.2.123.1:44875/ using document root <none> and password file <none>
I20260928 02:45:18.809281 19650 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:18.809348 19650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:18.809538 19650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:18.811611 19650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "e351c6ceacc0407d9d2b4cc78c0e1476"
format_stamp: "Formatted at 2026-09-28 02:45:18 on dist-test-slave-rqvb"
server_key: "997b584cbd58c038e22248b5b230e98d"
server_key_iv: "e97720f5873ea05e0e555853b4d0c013"
server_key_version: "encryptionkey@0"
I20260928 02:45:18.812158 19650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "e351c6ceacc0407d9d2b4cc78c0e1476"
format_stamp: "Formatted at 2026-09-28 02:45:18 on dist-test-slave-rqvb"
server_key: "997b584cbd58c038e22248b5b230e98d"
server_key_iv: "e97720f5873ea05e0e555853b4d0c013"
server_key_version: "encryptionkey@0"
I20260928 02:45:18.815970 19650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:45:18.818544 19665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:18.820035 19650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:18.820178 19650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "e351c6ceacc0407d9d2b4cc78c0e1476"
format_stamp: "Formatted at 2026-09-28 02:45:18 on dist-test-slave-rqvb"
server_key: "997b584cbd58c038e22248b5b230e98d"
server_key_iv: "e97720f5873ea05e0e555853b4d0c013"
server_key_version: "encryptionkey@0"
I20260928 02:45:18.820289 19650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:18.852067 19650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:18.855677 19650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:18.855870 19650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:18.856468 19650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:18.857300 19650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:18.857345 19650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:18.857395 19650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:18.857424 19650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:18.865989 19650 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:45663
I20260928 02:45:18.866074 19778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:45663 every 8 connection(s)
I20260928 02:45:18.866883 19650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:45:18.875111 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 19650
I20260928 02:45:18.875237 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:45:18.875550 2540 external_mini_cluster.cc:1428] Setting key b35172669772ea12c808629f981ac3a7
Sep 28 02:45:18 dist-test-slave-rqvb krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:18.880303 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:18.868740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:35737 (local address 127.2.123.62:46571)
0928 02:45:18.868901 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:18.868906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:18.870000 (+ 1094us) server_negotiation.cc:407] Connection header received
0928 02:45:18.870884 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:18.870888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:18.870960 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:18.871051 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:18.872240 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:18.872803 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:18.874039 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:18.874280 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:18.874370 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:18.876960 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:18.876963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:18.876966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:18.876994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:18.878319 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:18.878835 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:18.878838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:18.878839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:18.878913 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:18.879241 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:18.879244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:18.879246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:18.879424 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:18.879509 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:18.880003 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 02:45:18.880142 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:45:18.881165 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:18.869140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:46571 (local address 127.2.123.1:35737)
0928 02:45:18.869814 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:18.869867 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:18.870675 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:18.871238 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:18.871246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:18.871615 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:18.872096 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:18.872110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:18.872927 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:18.872930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:18.873896 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:18.873905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:18.874467 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:18.874470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:18.874622 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:18.875198 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:18.875216 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:18.876806 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:18.878432 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:18.878438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:18.878454 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:18.878737 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:18.878995 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:18.878998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:18.879000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:18.879154 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:18.879523 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:18.879530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:18.879780 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 02:45:18.879948 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 02:45:18.880304 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":115,"threads_started":1}
I20260928 02:45:18.882458 19779 heartbeater.cc:347] Connected to a master server at 127.2.123.62:46571
I20260928 02:45:18.882743 19779 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:18.883359 19779 heartbeater.cc:511] Master 127.2.123.62:46571 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:45:18.884965 19588 ts_manager.cc:194] Registered new tserver with Master: e351c6ceacc0407d9d2b4cc78c0e1476 (127.2.123.1:45663)
I20260928 02:45:18.886485 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:35737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:45:18.902985 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46571
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:19.012625 19784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:19.012882 19784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:19.012925 19784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:19.016902 19784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:19.016964 19784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:19.017026 19784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:45:19.022300 19784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:19.023306 19784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:19.024292 19784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:19.033895 19784 server_base.cc:1034] running on GCE node
W20260928 02:45:19.033833 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:19.033841 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:19.033850 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:19.034473 19784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:19.035123 19784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:19.036301 19784 hybrid_clock.cc:648] HybridClock initialized: now 1790563519036267 us; error 43 us; skew 500 ppm
Sep 28 02:45:19 dist-test-slave-rqvb krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563519, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:19.039202 19784 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:45:19.040127 19784 webserver.cc:460] Webserver started at http://127.2.123.2:42701/ using document root <none> and password file <none>
I20260928 02:45:19.040707 19784 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:19.040782 19784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:19.041002 19784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:19.043205 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "9b8f8393482a4f8d82364d01b95b8335"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "2e9cd1032cdca5a34e4b0955e4221e9f"
server_key_iv: "51710a3f313b520d232255d921db35dc"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.043804 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "9b8f8393482a4f8d82364d01b95b8335"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "2e9cd1032cdca5a34e4b0955e4221e9f"
server_key_iv: "51710a3f313b520d232255d921db35dc"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.047839 19784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:45:19.050561 19799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:19.051817 19784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:19.051927 19784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "9b8f8393482a4f8d82364d01b95b8335"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "2e9cd1032cdca5a34e4b0955e4221e9f"
server_key_iv: "51710a3f313b520d232255d921db35dc"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.052027 19784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:19.074719 19784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:19.077925 19784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:19.078127 19784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:19.078716 19784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:19.079538 19784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:19.079594 19784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:19.079648 19784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:19.079670 19784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:19.088385 19784 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:41485
I20260928 02:45:19.088411 19912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:41485 every 8 connection(s)
I20260928 02:45:19.089304 19784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:45:19.089612 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 19784
I20260928 02:45:19.089735 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:45:19.090109 2540 external_mini_cluster.cc:1428] Setting key 04b6fb2906f68f896461237fce0834b5
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
Sep 28 02:45:19 dist-test-slave-rqvb krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563519, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:45:19.103386 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.091260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:48035 (local address 127.2.123.62:46571)
0928 02:45:19.091398 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.091402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.092584 (+ 1182us) server_negotiation.cc:407] Connection header received
0928 02:45:19.093891 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.093894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.093948 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.094052 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:19.095227 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.095761 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.096882 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.097107 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.097192 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.099695 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:19.099703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:19.099705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:19.099730 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:19.101124 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:19.101703 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:19.101706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:19.101708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:19.101786 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:19.102158 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:19.102161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:19.102163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:19.102350 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:19.102439 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.103147 (+ 708us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.103261 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:45:19.104416 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.091893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:46571 (local address 127.2.123.2:48035)
0928 02:45:19.092426 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:19.092470 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:19.093689 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:19.094240 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:19.094247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:19.094604 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:19.095077 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.095102 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.095879 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.095881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.096765 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.096774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.097243 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.097247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.097405 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.097937 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:19.097952 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:19.099491 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:19.101278 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:19.101283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:19.101297 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:19.101594 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:19.101873 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:19.101876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:19.101877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:19.102065 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:19.102450 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:19.102457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:19.102859 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 02:45:19.103082 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 02:45:19.103445 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":152,"threads_started":1}
I20260928 02:45:19.105593 19913 heartbeater.cc:347] Connected to a master server at 127.2.123.62:46571
I20260928 02:45:19.105831 19913 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:19.106353 19913 heartbeater.cc:511] Master 127.2.123.62:46571 requested a full tablet report, sending...
I20260928 02:45:19.107424 19588 ts_manager.cc:194] Registered new tserver with Master: 9b8f8393482a4f8d82364d01b95b8335 (127.2.123.2:41485)
I20260928 02:45:19.108209 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:48035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:45:19.117498 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46571
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:19.227782 19918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:19.228085 19918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:19.228135 19918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:19.232300 19918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:19.232370 19918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:19.232432 19918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:45:19.237854 19918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:19.238940 19918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:19.239952 19918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:19.249622 19926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:19.249622 19923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:19.249691 19924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:19.249657 19918 server_base.cc:1034] running on GCE node
I20260928 02:45:19.250249 19918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:19.250871 19918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:19.252043 19918 hybrid_clock.cc:648] HybridClock initialized: now 1790563519252028 us; error 31 us; skew 500 ppm
Sep 28 02:45:19 dist-test-slave-rqvb krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563519, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:19.254628 19918 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:45:19.255474 19918 webserver.cc:460] Webserver started at http://127.2.123.3:35117/ using document root <none> and password file <none>
I20260928 02:45:19.256014 19918 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:19.256078 19918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:19.256246 19918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:19.258279 19918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "c4d68f2d53dd4018bda142f80a463699"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "bbb9c0c9e5510dd84efed991a9aadafd"
server_key_iv: "11f5d72c657a570a36a48c1c52720f77"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.258798 19918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "c4d68f2d53dd4018bda142f80a463699"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "bbb9c0c9e5510dd84efed991a9aadafd"
server_key_iv: "11f5d72c657a570a36a48c1c52720f77"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.262459 19918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:45:19.264915 19933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:19.266335 19918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:19.266440 19918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "c4d68f2d53dd4018bda142f80a463699"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "bbb9c0c9e5510dd84efed991a9aadafd"
server_key_iv: "11f5d72c657a570a36a48c1c52720f77"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.266567 19918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:19.297518 19918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:19.300871 19918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:19.301067 19918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:19.301681 19918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:19.302546 19918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:19.302594 19918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:19.302649 19918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:19.302675 19918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:19.311676 19918 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:44955
I20260928 02:45:19.311717 20046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:44955 every 8 connection(s)
I20260928 02:45:19.312763 19918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:45:19.314206 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 19918
I20260928 02:45:19.314328 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:45:19.314705 2540 external_mini_cluster.cc:1428] Setting key 9193eae3cf7b27f264d4f3bb8380f0d7
Sep 28 02:45:19 dist-test-slave-rqvb krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563519, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:19.326162 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.315074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:46263 (local address 127.2.123.62:46571)
0928 02:45:19.315217 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.315221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.315879 (+ 658us) server_negotiation.cc:407] Connection header received
0928 02:45:19.316822 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.316824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.316868 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.316933 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:19.318125 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.318693 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.319919 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.320167 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.320254 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.322777 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:19.322780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:19.322782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:19.322827 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:19.324142 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:19.324668 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:19.324672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:19.324675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:19.324756 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:19.325115 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:19.325117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:19.325118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:19.325312 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:19.325400 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.325878 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.326032 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:45:19.326988 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.315051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:46571 (local address 127.2.123.3:46263)
0928 02:45:19.315614 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:19.315677 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:19.316640 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:19.317069 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:19.317076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:19.317433 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:19.317992 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.318014 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.318832 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.318835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.319791 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.319802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.320290 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.320293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.320434 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.321093 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:19.321109 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:19.322608 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:19.324250 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:19.324256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:19.324274 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:19.324543 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:19.324825 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:19.324828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:19.324829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:19.324960 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:19.325405 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:19.325410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:19.325665 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 02:45:19.325823 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 02:45:19.326136 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":143,"threads_started":1}
I20260928 02:45:19.328209 20047 heartbeater.cc:347] Connected to a master server at 127.2.123.62:46571
I20260928 02:45:19.328447 20047 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:19.328981 20047 heartbeater.cc:511] Master 127.2.123.62:46571 requested a full tablet report, sending...
I20260928 02:45:19.330073 19588 ts_manager.cc:194] Registered new tserver with Master: c4d68f2d53dd4018bda142f80a463699 (127.2.123.3:44955)
I20260928 02:45:19.330649 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:46263
I20260928 02:45:19.340528 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:45:19.349730 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.342519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.2.123.62:46571)
0928 02:45:19.342651 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.342656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.342950 (+ 294us) server_negotiation.cc:407] Connection header received
0928 02:45:19.342996 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.342998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.343046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.343118 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:19.343766 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.344311 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.345107 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.345318 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.345394 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.346713 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:19.346716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:19.346719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:19.346742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:19.348114 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:19.348567 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:19.348570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:19.348571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:19.348687 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:19.348905 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:19.348907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:19.348908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:19.349115 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:19.349221 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.349447 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.349543 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:45:19.352680 19588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:45:19.355751 19588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:45:19.370008 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.363986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:44955 (local address 127.0.0.1:51448)
0928 02:45:19.364809 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:19.364860 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:19.364990 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:19.365887 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:19.365893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:19.365924 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:19.366140 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.366146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.367247 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.367250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.368541 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.368551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.369418 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.369421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.369526 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.369569 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:45:19.369690 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 02:45:19.369809 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":124,"spinlock_wait_cycles":9216,"thread_start_us":91,"threads_started":1}
I20260928 02:45:19.370016 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.364155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.2.123.3:44955)
0928 02:45:19.365520 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.365532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.365550 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:45:19.365605 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.365608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.365748 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.365875 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:19.366258 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.367135 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.368663 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.369296 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.369406 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.369561 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.369730 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.369829 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1283,"thread_start_us":110,"threads_started":1}
I20260928 02:45:19.370621 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.364742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:41485 (local address 127.0.0.1:34872)
0928 02:45:19.365228 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:19.365240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:19.365315 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:19.365887 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:19.365893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:19.365928 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:19.366136 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.366144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.367634 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.367637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.368977 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.368987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.370217 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.370220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.370313 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.370335 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:45:19.370415 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:45:19.370483 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":11,"spinlock_wait_cycles":5376,"thread_start_us":80,"threads_started":1}
I20260928 02:45:19.370817 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.364325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.2.123.1:45663)
0928 02:45:19.366212 (+ 1887us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.366227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.366250 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:45:19.366320 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.366324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.366514 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.366662 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:19.366865 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.367934 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.369269 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.370105 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.370361 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.370494 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.370578 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.370667 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":79,"threads_started":1}
I20260928 02:45:19.371153 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.365111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.2.123.2:41485)
0928 02:45:19.365416 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.365425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.365444 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:45:19.365506 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.365510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.365739 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.365875 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:19.366276 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.367449 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.369100 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.369883 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.370005 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.370153 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.370890 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.370988 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260928 02:45:19.370872 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.364131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:45663 (local address 127.0.0.1:47198)
0928 02:45:19.364807 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:19.364860 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:19.364990 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:19.366655 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:19.366658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:19.366676 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:19.366755 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.366760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.368071 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.368074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.369160 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.369167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.370374 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.370377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.370455 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.370475 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:45:19.370548 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:45:19.370604 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"spinlock_wait_cycles":2560,"thread_start_us":115,"threads_started":1}
I20260928 02:45:19.373322 19981 tablet_service.cc:1467] Processing CreateTablet for tablet 959a1ac0fc4941218650daac7f43f37d (DEFAULT_TABLE table=test-table [id=578e288ffe684a6381da3ad22bc27433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:19.373732 19847 tablet_service.cc:1467] Processing CreateTablet for tablet 959a1ac0fc4941218650daac7f43f37d (DEFAULT_TABLE table=test-table [id=578e288ffe684a6381da3ad22bc27433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:19.374322 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959a1ac0fc4941218650daac7f43f37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:19.374791 19847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959a1ac0fc4941218650daac7f43f37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:19.376573 19713 tablet_service.cc:1467] Processing CreateTablet for tablet 959a1ac0fc4941218650daac7f43f37d (DEFAULT_TABLE table=test-table [id=578e288ffe684a6381da3ad22bc27433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:19.377653 19713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959a1ac0fc4941218650daac7f43f37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:19.380831 20064 tablet_bootstrap.cc:492] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Bootstrap starting.
I20260928 02:45:19.382063 20065 tablet_bootstrap.cc:492] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Bootstrap starting.
I20260928 02:45:19.383901 20064 tablet_bootstrap.cc:654] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:19.384286 20065 tablet_bootstrap.cc:654] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:19.384640 20064 log.cc:824] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:19.385423 20065 log.cc:824] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:19.385291 20066 tablet_bootstrap.cc:492] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476: Bootstrap starting.
I20260928 02:45:19.386571 20064 tablet_bootstrap.cc:492] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: No bootstrap required, opened a new log
I20260928 02:45:19.386785 20064 ts_tablet_manager.cc:1397] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 02:45:19.387116 20065 tablet_bootstrap.cc:492] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: No bootstrap required, opened a new log
I20260928 02:45:19.387291 20065 ts_tablet_manager.cc:1397] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:45:19.388088 20066 tablet_bootstrap.cc:654] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:19.389107 20066 log.cc:824] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:19.390538 20065 raft_consensus.cc:348] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.390508 20064 raft_consensus.cc:348] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.390769 20065 raft_consensus.cc:374] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:19.390771 20064 raft_consensus.cc:374] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:19.390825 20065 raft_consensus.cc:729] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d68f2d53dd4018bda142f80a463699, State: Initialized, Role: FOLLOWER
I20260928 02:45:19.390825 20064 raft_consensus.cc:729] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8f8393482a4f8d82364d01b95b8335, State: Initialized, Role: FOLLOWER
I20260928 02:45:19.391144 20066 tablet_bootstrap.cc:492] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476: No bootstrap required, opened a new log
I20260928 02:45:19.391240 20065 consensus_queue.cc:260] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.391323 20066 ts_tablet_manager.cc:1397] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 02:45:19.391279 20064 consensus_queue.cc:260] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.391855 20047 heartbeater.cc:503] Master 127.2.123.62:46571 was elected leader, sending a full tablet report...
I20260928 02:45:19.392174 20065 ts_tablet_manager.cc:1428] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:45:19.392202 20064 ts_tablet_manager.cc:1428] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:45:19.392493 19913 heartbeater.cc:503] Master 127.2.123.62:46571 was elected leader, sending a full tablet report...
I20260928 02:45:19.396687 20066 raft_consensus.cc:348] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.396912 20066 raft_consensus.cc:374] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:19.396961 20066 raft_consensus.cc:729] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e351c6ceacc0407d9d2b4cc78c0e1476, State: Initialized, Role: FOLLOWER
I20260928 02:45:19.397363 20066 consensus_queue.cc:260] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.397951 19779 heartbeater.cc:503] Master 127.2.123.62:46571 was elected leader, sending a full tablet report...
I20260928 02:45:19.398239 20066 ts_tablet_manager.cc:1428] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
W20260928 02:45:19.563710 20048 tablet.cc:2369] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:19.583734 20071 raft_consensus.cc:482] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:19.583942 20071 raft_consensus.cc:504] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.585089 20071 leader_election.cc:288] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d68f2d53dd4018bda142f80a463699 (127.2.123.3:44955), e351c6ceacc0407d9d2b4cc78c0e1476 (127.2.123.1:45663)
I20260928 02:45:19.588939 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.585365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:44955 (local address 127.2.123.2:56289)
0928 02:45:19.585467 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:19.585482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:19.585575 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:19.585829 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:19.585832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:19.585857 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:19.585987 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.585995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.586852 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.586862 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.587747 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.587756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.588479 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.588483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.588651 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.588677 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:45:19.588746 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:45:19.588818 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 02:45:19.589020 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.585434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:56289 (local address 127.2.123.3:44955)
0928 02:45:19.585565 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.585569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.585585 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:45:19.585682 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.585684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.585728 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.585798 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:19.586122 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.586741 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.587868 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.588366 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.588440 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.588498 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.588776 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.588851 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:45:19.589211 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.585368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:45663 (local address 127.2.123.2:59485)
0928 02:45:19.585899 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:19.585911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:19.586030 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:19.586237 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:19.586240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:19.586258 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:19.586383 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.586389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.587213 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.587216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.588074 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:19.588084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.588786 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.588789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:19.588884 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.588907 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:45:19.588983 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:45:19.589048 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":279,"thread_start_us":87,"threads_started":1}
I20260928 02:45:19.589205 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.585477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:59485 (local address 127.2.123.1:45663)
0928 02:45:19.585603 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.585608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.586036 (+ 428us) server_negotiation.cc:407] Connection header received
0928 02:45:19.586108 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.586111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.586152 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.586207 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:19.586480 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.587089 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.588189 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.588655 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.588758 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.588813 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.589015 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.589092 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:45:19.589818 19733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959a1ac0fc4941218650daac7f43f37d" candidate_uuid: "9b8f8393482a4f8d82364d01b95b8335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" is_pre_election: true
I20260928 02:45:19.589790 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959a1ac0fc4941218650daac7f43f37d" candidate_uuid: "9b8f8393482a4f8d82364d01b95b8335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d68f2d53dd4018bda142f80a463699" is_pre_election: true
I20260928 02:45:19.590163 20001 raft_consensus.cc:2454] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b8f8393482a4f8d82364d01b95b8335 in term 0.
I20260928 02:45:19.590163 19733 raft_consensus.cc:2454] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b8f8393482a4f8d82364d01b95b8335 in term 0.
I20260928 02:45:19.590700 19801 leader_election.cc:302] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8f8393482a4f8d82364d01b95b8335, c4d68f2d53dd4018bda142f80a463699; no voters:
W20260928 02:45:19.590750 19914 tablet.cc:2369] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:19.590998 20071 raft_consensus.cc:2781] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:45:19.591112 20071 raft_consensus.cc:482] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:45:19.591164 20071 raft_consensus.cc:3037] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:19.592196 20071 raft_consensus.cc:504] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.592543 20071 leader_election.cc:288] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [CANDIDATE]: Term 1 election: Requested vote from peers c4d68f2d53dd4018bda142f80a463699 (127.2.123.3:44955), e351c6ceacc0407d9d2b4cc78c0e1476 (127.2.123.1:45663)
I20260928 02:45:19.592895 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959a1ac0fc4941218650daac7f43f37d" candidate_uuid: "9b8f8393482a4f8d82364d01b95b8335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d68f2d53dd4018bda142f80a463699"
I20260928 02:45:19.592893 19733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959a1ac0fc4941218650daac7f43f37d" candidate_uuid: "9b8f8393482a4f8d82364d01b95b8335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476"
I20260928 02:45:19.593006 20001 raft_consensus.cc:3037] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:19.593012 19733 raft_consensus.cc:3037] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:19.594051 20001 raft_consensus.cc:2454] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b8f8393482a4f8d82364d01b95b8335 in term 1.
I20260928 02:45:19.594151 19733 raft_consensus.cc:2454] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b8f8393482a4f8d82364d01b95b8335 in term 1.
I20260928 02:45:19.594329 19801 leader_election.cc:302] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8f8393482a4f8d82364d01b95b8335, c4d68f2d53dd4018bda142f80a463699; no voters:
I20260928 02:45:19.594482 20071 raft_consensus.cc:2781] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:19.594724 20071 raft_consensus.cc:686] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 1 LEADER]: Becoming Leader. State: Replica: 9b8f8393482a4f8d82364d01b95b8335, State: Running, Role: LEADER
I20260928 02:45:19.595021 20071 consensus_queue.cc:237] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } }
I20260928 02:45:19.598119 19586 catalog_manager.cc:5495] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b8f8393482a4f8d82364d01b95b8335 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "9b8f8393482a4f8d82364d01b95b8335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b8f8393482a4f8d82364d01b95b8335" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 41485 } health_report { overall_health: HEALTHY } } }
W20260928 02:45:19.618223 19780 tablet.cc:2369] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:19.663911 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.660422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.2.123.2:41485)
0928 02:45:19.660544 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.660548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.660632 (+ 84us) server_negotiation.cc:407] Connection header received
0928 02:45:19.660724 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.660728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.660786 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.660886 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:19.661106 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.661657 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.662520 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.662789 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.662869 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.663114 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:19.663523 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:19.663675 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.663730 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.663778 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:45:19.669723 20001 raft_consensus.cc:1261] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 1 FOLLOWER]: Refusing update from remote peer 9b8f8393482a4f8d82364d01b95b8335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:19.669749 19733 raft_consensus.cc:1261] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476 [term 1 FOLLOWER]: Refusing update from remote peer 9b8f8393482a4f8d82364d01b95b8335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:19.670493 20074 consensus_queue.cc:1035] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d68f2d53dd4018bda142f80a463699" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 44955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:19.670688 20071 consensus_queue.cc:1035] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e351c6ceacc0407d9d2b4cc78c0e1476" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:19.677536 20077 mvcc.cc:205] Tried to move back new op lower bound from 7334148176560168960 to 7334148176263729152. Current Snapshot: MvccSnapshot[applied={T|T < 7334148176560168960}]
I20260928 02:45:19.678876 20079 mvcc.cc:205] Tried to move back new op lower bound from 7334148176560168960 to 7334148176263729152. Current Snapshot: MvccSnapshot[applied={T|T < 7334148176560168960}]
I20260928 02:45:19.679147 20078 mvcc.cc:205] Tried to move back new op lower bound from 7334148176560168960 to 7334148176263729152. Current Snapshot: MvccSnapshot[applied={T|T < 7334148176560168960}]
I20260928 02:45:19.685010 19588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:19.685206 19588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:19.687575 19588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62bc9c6c35034473860193684183fb52: Sending DeleteTablet for 3 replicas of tablet 959a1ac0fc4941218650daac7f43f37d
I20260928 02:45:19.688385 19981 tablet_service.cc:1514] Processing DeleteTablet for tablet 959a1ac0fc4941218650daac7f43f37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:19 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:51448
I20260928 02:45:19.688365 19713 tablet_service.cc:1514] Processing DeleteTablet for tablet 959a1ac0fc4941218650daac7f43f37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:19 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:47198
I20260928 02:45:19.688517 19847 tablet_service.cc:1514] Processing DeleteTablet for tablet 959a1ac0fc4941218650daac7f43f37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:19 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:34872
I20260928 02:45:19.688928 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 19650
I20260928 02:45:19.689015 20087 tablet_replica.cc:330] T 959a1ac0fc4941218650daac7f43f37d P e351c6ceacc0407d9d2b4cc78c0e1476: stopping tablet replica
I20260928 02:45:19.689570 20088 tablet_replica.cc:330] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: stopping tablet replica
I20260928 02:45:19.689836 20088 raft_consensus.cc:2229] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:45:19.690176 20086 tablet_replica.cc:330] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: stopping tablet replica
I20260928 02:45:19.690256 20088 raft_consensus.cc:2258] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:19.690425 20086 raft_consensus.cc:2229] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:19.690711 20086 raft_consensus.cc:2258] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:19.692250 20086 ts_tablet_manager.cc:1905] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:19.693048 20088 ts_tablet_manager.cc:1905] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:19.695014 20086 ts_tablet_manager.cc:1918] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:19.695119 20086 log.cc:1196] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/959a1ac0fc4941218650daac7f43f37d
I20260928 02:45:19.695432 20086 ts_tablet_manager.cc:1939] T 959a1ac0fc4941218650daac7f43f37d P c4d68f2d53dd4018bda142f80a463699: Deleting consensus metadata
I20260928 02:45:19.695606 20088 ts_tablet_manager.cc:1918] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:19.695708 20088 log.cc:1196] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/959a1ac0fc4941218650daac7f43f37d
I20260928 02:45:19.695984 20088 ts_tablet_manager.cc:1939] T 959a1ac0fc4941218650daac7f43f37d P 9b8f8393482a4f8d82364d01b95b8335: Deleting consensus metadata
I20260928 02:45:19.696693 19573 catalog_manager.cc:4841] TS c4d68f2d53dd4018bda142f80a463699 (127.2.123.3:44955): tablet 959a1ac0fc4941218650daac7f43f37d (table test-table [id=578e288ffe684a6381da3ad22bc27433]) successfully deleted
I20260928 02:45:19.697191 19573 catalog_manager.cc:4841] TS 9b8f8393482a4f8d82364d01b95b8335 (127.2.123.2:41485): tablet 959a1ac0fc4941218650daac7f43f37d (table test-table [id=578e288ffe684a6381da3ad22bc27433]) successfully deleted
W20260928 02:45:19.697250 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:45663 (error 108)
I20260928 02:45:19.697561 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 19784
I20260928 02:45:19.698249 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.698001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:45663 (local address 127.0.0.1:47202)
0928 02:45:19.698099 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:19.698166 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:45:19.698446 19572 catalog_manager.cc:4584] TS e351c6ceacc0407d9d2b4cc78c0e1476 (127.2.123.1:45663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 959a1ac0fc4941218650daac7f43f37d: Network error: Client connection negotiation failed: client connection to 127.2.123.1:45663: connect: Connection refused (error 111)
I20260928 02:45:19.702942 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 19918
I20260928 02:45:19.707434 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 19556
2026-09-28T02:45:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2405 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:19Z Disabled control of system clock
I20260928 02:45:19.735616 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:43813
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:43813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:45:19.844666 20093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:19.844972 20093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:19.845007 20093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:19.849118 20093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:19.849203 20093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:19.849218 20093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:19.849233 20093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:19.849248 20093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:19.854184 20093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:43813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:19.855378 20093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:19.856817 20093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:19.865437 20099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:19.865463 20101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:19.865504 20093 server_base.cc:1034] running on GCE node
W20260928 02:45:19.865463 20098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:19.866411 20093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:19.867472 20093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:19.868639 20093 hybrid_clock.cc:648] HybridClock initialized: now 1790563519868613 us; error 42 us; skew 500 ppm
I20260928 02:45:19.870559 20093 webserver.cc:460] Webserver started at http://127.2.123.62:36453/ using document root <none> and password file <none>
I20260928 02:45:19.871143 20093 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:19.871232 20093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:19.871469 20093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:19.874277 20093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "3f9a1ba77cc54bf685a4ecfa060340b0"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "e8b70a5fefe4bb69c3f300c6eea15652"
server_key_iv: "73e27a2b71b866efe3aa78e05c85fd56"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.874997 20093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "3f9a1ba77cc54bf685a4ecfa060340b0"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "e8b70a5fefe4bb69c3f300c6eea15652"
server_key_iv: "73e27a2b71b866efe3aa78e05c85fd56"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.879302 20093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:45:19.882017 20107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:19.883237 20093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:19.883345 20093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "3f9a1ba77cc54bf685a4ecfa060340b0"
format_stamp: "Formatted at 2026-09-28 02:45:19 on dist-test-slave-rqvb"
server_key: "e8b70a5fefe4bb69c3f300c6eea15652"
server_key_iv: "73e27a2b71b866efe3aa78e05c85fd56"
server_key_version: "encryptionkey@0"
I20260928 02:45:19.883453 20093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:19.894912 20093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:19.895587 20093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:19.895753 20093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:19.902190 20093 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:43813
I20260928 02:45:19.902209 20159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:43813 every 8 connection(s)
I20260928 02:45:19.903128 20093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:19.906208 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:19.911238 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 20093
I20260928 02:45:19.911356 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:19.911643 2540 external_mini_cluster.cc:1428] Setting key c29d2075c5ce9143e9d92aecc48b7c78
I20260928 02:45:19.911739 20160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0: Bootstrap starting.
I20260928 02:45:19.914407 20160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:19.915134 20160 log.cc:824] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:19.916996 20160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0: No bootstrap required, opened a new log
I20260928 02:45:19.920711 20160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a1ba77cc54bf685a4ecfa060340b0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43813 } }
I20260928 02:45:19.920956 20160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:19.921010 20160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9a1ba77cc54bf685a4ecfa060340b0, State: Initialized, Role: FOLLOWER
I20260928 02:45:19.921190 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:19.913167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.2.123.62:43813)
0928 02:45:19.913858 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:19.913872 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:19.913907 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:45:19.914676 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:19.914695 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:19.915128 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:19.915474 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:19.915898 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.916881 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.917838 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:19.918385 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:19.918540 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:19.919364 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:19.919370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:19.919383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:19.919808 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:19.919863 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:19.919884 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:19.919968 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:19.920112 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:45:19.920390 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260928 02:45:19.921479 20160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a1ba77cc54bf685a4ecfa060340b0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43813 } }
I20260928 02:45:19.921635 20160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:19.921703 20160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:19.921795 20160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:19.922838 20160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a1ba77cc54bf685a4ecfa060340b0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43813 } }
I20260928 02:45:19.923170 20160 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9a1ba77cc54bf685a4ecfa060340b0; no voters:
I20260928 02:45:19.923496 20160 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:19.923750 20165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:19.924050 20165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3f9a1ba77cc54bf685a4ecfa060340b0, State: Running, Role: LEADER
I20260928 02:45:19.924371 20165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a1ba77cc54bf685a4ecfa060340b0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43813 } }
I20260928 02:45:19.924707 20160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:19.925901 20166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f9a1ba77cc54bf685a4ecfa060340b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a1ba77cc54bf685a4ecfa060340b0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43813 } } }
I20260928 02:45:19.926064 20166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:19.926374 20167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9a1ba77cc54bf685a4ecfa060340b0. Latest consensus state: current_term: 1 leader_uuid: "3f9a1ba77cc54bf685a4ecfa060340b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a1ba77cc54bf685a4ecfa060340b0" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 43813 } } }
I20260928 02:45:19.926502 20167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0 [sys.catalog]: This master's current role is: LEADER
W20260928 02:45:19.929296 20180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:19.929378 20180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:19.929474 20175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:19.930590 20175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:19.936201 20175 catalog_manager.cc:1347] Generated new cluster ID: e3c8348781bd4ab587dd669fdeea7614
I20260928 02:45:19.936280 20175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:19.948055 20175 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:19.949064 20175 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:19.957578 20175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0: Generated new TSK 0
I20260928 02:45:19.958460 20175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:19.972939 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43813
--builtin_ntp_servers=127.2.123.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:45:20.081897 20184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:20.082201 20184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:20.082257 20184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:20.086292 20184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:20.086347 20184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:20.086419 20184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:45:20.093235 20184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:20.094465 20184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:20.096055 20184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:20.105633 20192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:20.105718 20190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:20.105644 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:20.105896 20184 server_base.cc:1034] running on GCE node
I20260928 02:45:20.106339 20184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:20.106966 20184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:20.108152 20184 hybrid_clock.cc:648] HybridClock initialized: now 1790563520108130 us; error 33 us; skew 500 ppm
I20260928 02:45:20.109938 20184 webserver.cc:460] Webserver started at http://127.2.123.1:33377/ using document root <none> and password file <none>
I20260928 02:45:20.110524 20184 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:20.110592 20184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:20.110787 20184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:20.112787 20184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "8786bd4d7d914574ba439b8b16a1a8a1"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "a1739053cb18d894a63712a2c9e038e8"
server_key_iv: "4ba543929c1ac46d445dd33537446b07"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.113328 20184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "8786bd4d7d914574ba439b8b16a1a8a1"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "a1739053cb18d894a63712a2c9e038e8"
server_key_iv: "4ba543929c1ac46d445dd33537446b07"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.116820 20184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260928 02:45:20.119228 20198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.120657 20184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:45:20.120761 20184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "8786bd4d7d914574ba439b8b16a1a8a1"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "a1739053cb18d894a63712a2c9e038e8"
server_key_iv: "4ba543929c1ac46d445dd33537446b07"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.120862 20184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:20.144470 20184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:20.145216 20184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:20.145397 20184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:20.146034 20184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:20.147003 20184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:20.147059 20184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.147111 20184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:20.147135 20184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.155340 20184 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:38875
I20260928 02:45:20.155422 20311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:38875 every 8 connection(s)
I20260928 02:45:20.156263 20184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:45:20.159518 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 20184
I20260928 02:45:20.159621 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:45:20.159919 2540 external_mini_cluster.cc:1428] Setting key 8b59ba79e132f2be8c1d3888e3ca12c2
I20260928 02:45:20.161917 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43813
--builtin_ntp_servers=127.2.123.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:45:20.166891 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.158101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:50569 (local address 127.2.123.62:43813)
0928 02:45:20.158233 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.158237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.159343 (+ 1106us) server_negotiation.cc:407] Connection header received
0928 02:45:20.160975 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.160979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.161036 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.161117 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:20.162404 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.163143 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.164769 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.165079 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.165185 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.166366 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:20.166370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:20.166373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:20.166390 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:20.166416 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:20.166427 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:20.166506 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.166663 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.166733 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:45:20.167754 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.158470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:43813 (local address 127.2.123.1:50569)
0928 02:45:20.159168 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.159211 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:20.160752 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:20.161318 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:20.161327 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:20.161440 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:20.162245 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.162264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.163289 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.163293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.164634 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.164644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.165247 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.165251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.165416 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.166082 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:20.166083 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:20.166133 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:20.166202 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:20.166517 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:20.166523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:20.166528 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:45:20.166635 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:45:20.166873 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260928 02:45:20.169062 20312 heartbeater.cc:347] Connected to a master server at 127.2.123.62:43813
I20260928 02:45:20.169346 20312 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:20.170044 20312 heartbeater.cc:511] Master 127.2.123.62:43813 requested a full tablet report, sending...
I20260928 02:45:20.171882 20124 ts_manager.cc:194] Registered new tserver with Master: 8786bd4d7d914574ba439b8b16a1a8a1 (127.2.123.1:38875)
I20260928 02:45:20.173482 20124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.1:50569
W20260928 02:45:20.273407 20315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:20.273693 20315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:20.273741 20315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:20.277750 20315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:20.277817 20315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:20.277892 20315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:45:20.282811 20315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:20.283911 20315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:20.285298 20315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:20.295463 20321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:20.295463 20320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:20.295650 20315 server_base.cc:1034] running on GCE node
W20260928 02:45:20.295537 20323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:20.296161 20315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:20.296967 20315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:20.298134 20315 hybrid_clock.cc:648] HybridClock initialized: now 1790563520298077 us; error 68 us; skew 500 ppm
I20260928 02:45:20.300112 20315 webserver.cc:460] Webserver started at http://127.2.123.2:33389/ using document root <none> and password file <none>
I20260928 02:45:20.300719 20315 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:20.300789 20315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:20.300988 20315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:20.303277 20315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "0e7e7d30c88e415faef6d5001a9f5647"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "270de6980991c5993ab9183be395701d"
server_key_iv: "09d31be094fafa5e972a73cdf1b92245"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.303867 20315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "0e7e7d30c88e415faef6d5001a9f5647"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "270de6980991c5993ab9183be395701d"
server_key_iv: "09d31be094fafa5e972a73cdf1b92245"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.307839 20315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 02:45:20.310562 20329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.312247 20315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:20.312378 20315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "0e7e7d30c88e415faef6d5001a9f5647"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "270de6980991c5993ab9183be395701d"
server_key_iv: "09d31be094fafa5e972a73cdf1b92245"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.312496 20315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:20.333334 20315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:20.334105 20315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:20.334281 20315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:20.334895 20315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:20.335855 20315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:20.335901 20315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.335953 20315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:20.335983 20315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.344328 20315 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:46639
I20260928 02:45:20.344398 20442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:46639 every 8 connection(s)
I20260928 02:45:20.345248 20315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:45:20.348973 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 20315
I20260928 02:45:20.349077 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:45:20.349344 2540 external_mini_cluster.cc:1428] Setting key 0d27ccb223bbefb310933211c9bf5a37
I20260928 02:45:20.351192 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43813
--builtin_ntp_servers=127.2.123.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:45:20.354598 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.347102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:59807 (local address 127.2.123.62:43813)
0928 02:45:20.347243 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.347247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.348332 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 02:45:20.349814 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.349817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.349875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.349951 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:20.350946 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.351553 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.352714 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.352935 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.353023 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.354123 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:20.354127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:20.354129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:20.354141 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:20.354159 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:20.354166 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:20.354224 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.354392 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.354454 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:45:20.355494 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.347526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:43813 (local address 127.2.123.2:59807)
0928 02:45:20.348191 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.348233 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:20.349606 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:20.350204 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:20.350212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:20.350305 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:20.350806 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.350819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.351678 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.351681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.352592 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.352602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.353078 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.353082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.353253 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.353828 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:20.353830 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:20.353873 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:20.353940 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:20.354235 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:20.354252 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:20.354258 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:45:20.354380 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:45:20.354709 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260928 02:45:20.356825 20443 heartbeater.cc:347] Connected to a master server at 127.2.123.62:43813
I20260928 02:45:20.357131 20443 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:20.357797 20443 heartbeater.cc:511] Master 127.2.123.62:43813 requested a full tablet report, sending...
I20260928 02:45:20.359022 20124 ts_manager.cc:194] Registered new tserver with Master: 0e7e7d30c88e415faef6d5001a9f5647 (127.2.123.2:46639)
I20260928 02:45:20.359603 20124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.2:59807
W20260928 02:45:20.460413 20446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:20.460683 20446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:20.460737 20446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:20.464973 20446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:20.465039 20446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:20.465147 20446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:45:20.469983 20446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:20.471048 20446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:20.472415 20446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:20.481922 20452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:20.481992 20446 server_base.cc:1034] running on GCE node
W20260928 02:45:20.481931 20451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:20.482017 20454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:20.482612 20446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:20.483265 20446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:20.484434 20446 hybrid_clock.cc:648] HybridClock initialized: now 1790563520484408 us; error 40 us; skew 500 ppm
I20260928 02:45:20.486296 20446 webserver.cc:460] Webserver started at http://127.2.123.3:38483/ using document root <none> and password file <none>
I20260928 02:45:20.486857 20446 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:20.486914 20446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:20.487090 20446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:20.489166 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "bb31dda358d6464db5a3dd8b14011ed0"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "024e219995bfcc475978703c914b5f6d"
server_key_iv: "e27b80cba30d2bdec9a5006710e53554"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.489657 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "bb31dda358d6464db5a3dd8b14011ed0"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "024e219995bfcc475978703c914b5f6d"
server_key_iv: "e27b80cba30d2bdec9a5006710e53554"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.493279 20446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:45:20.495783 20460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.496963 20446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:20.497069 20446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "bb31dda358d6464db5a3dd8b14011ed0"
format_stamp: "Formatted at 2026-09-28 02:45:20 on dist-test-slave-rqvb"
server_key: "024e219995bfcc475978703c914b5f6d"
server_key_iv: "e27b80cba30d2bdec9a5006710e53554"
server_key_version: "encryptionkey@0"
I20260928 02:45:20.497167 20446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:20.507450 20446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:20.508131 20446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:20.508308 20446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:20.509078 20446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:20.510090 20446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:20.510146 20446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.510200 20446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:20.510227 20446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:20.518545 20446 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:41589
I20260928 02:45:20.518581 20573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:41589 every 8 connection(s)
I20260928 02:45:20.519441 20446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:45:20.527137 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 20446
I20260928 02:45:20.527230 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:45:20.527467 2540 external_mini_cluster.cc:1428] Setting key 28640bb3bf95e66d73525a16bb617547
I20260928 02:45:20.528980 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.521322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:51253 (local address 127.2.123.62:43813)
0928 02:45:20.521488 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.521491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.522563 (+ 1072us) server_negotiation.cc:407] Connection header received
0928 02:45:20.523972 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.523977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.524086 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.524176 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:20.525138 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.525716 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.526989 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.527256 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.527354 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.528466 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:20.528470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:20.528472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:20.528485 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:20.528507 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:20.528517 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:20.528587 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.528779 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.528839 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:45:20.529938 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.521705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:43813 (local address 127.2.123.3:51253)
0928 02:45:20.522431 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.522473 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:20.523711 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:20.524341 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:20.524348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:20.524435 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:20.524933 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.524945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.525843 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.525846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.526880 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.526890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.527400 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.527403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.527581 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.528187 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:20.528189 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:20.528231 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:20.528314 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:20.528598 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:20.528606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:20.528612 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:45:20.528748 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 02:45:20.529022 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":128,"threads_started":1}
I20260928 02:45:20.531073 20574 heartbeater.cc:347] Connected to a master server at 127.2.123.62:43813
I20260928 02:45:20.531306 20574 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:20.531795 20574 heartbeater.cc:511] Master 127.2.123.62:43813 requested a full tablet report, sending...
I20260928 02:45:20.532763 20124 ts_manager.cc:194] Registered new tserver with Master: bb31dda358d6464db5a3dd8b14011ed0 (127.2.123.3:41589)
I20260928 02:45:20.533314 20124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.3:51253
I20260928 02:45:20.541318 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:45:20.546849 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.543225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.2.123.62:43813)
0928 02:45:20.543391 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.543396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.543560 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:45:20.543653 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.543655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.543699 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.543772 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:20.544013 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.544557 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.545361 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.545635 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.545728 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.546466 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:20.546469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:20.546472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:20.546482 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:20.546501 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:20.546509 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:20.546572 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.546653 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.546703 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:45:20.549564 20124 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:45:20.552723 20124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:45:20.566891 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.561200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:38875 (local address 127.0.0.1:39364)
0928 02:45:20.561642 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.561690 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:20.561809 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:20.562521 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:20.562527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:20.562552 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:20.562689 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.562696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.564006 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.564009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.565324 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.565334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.566350 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.566354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.566457 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.566494 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:45:20.566608 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:45:20.566733 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260928 02:45:20.566898 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.561392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.2.123.1:38875)
0928 02:45:20.561886 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.561900 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.561926 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:45:20.562037 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.562043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.562270 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.562447 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:20.562811 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.563909 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.565461 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.566237 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.566338 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.566545 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.566637 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.566733 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":150,"threads_started":1}
I20260928 02:45:20.567436 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.561356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:41589 (local address 127.0.0.1:51296)
0928 02:45:20.562482 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.562498 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:20.562596 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:20.563685 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:20.563688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:20.563706 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:20.563790 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.563796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.564763 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.564767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.565827 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.565838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.566974 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.566978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.567160 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.567186 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:45:20.567235 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:45:20.567300 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":41,"spinlock_wait_cycles":3840,"thread_start_us":110,"threads_started":1}
I20260928 02:45:20.567924 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.561903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:46639 (local address 127.0.0.1:44364)
0928 02:45:20.562910 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.562922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:20.563008 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:20.563401 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:20.563404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:20.563419 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:20.563500 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.563506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.564819 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.564822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.565927 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.565936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.567544 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.567547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.567644 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.567668 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:45:20.567737 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:45:20.567806 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":124,"threads_started":1}
I20260928 02:45:20.568228 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.562165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.2.123.2:46639)
0928 02:45:20.562870 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.562883 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.563059 (+ 176us) server_negotiation.cc:407] Connection header received
0928 02:45:20.563133 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.563137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.563286 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.563410 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:20.563599 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.564680 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.566073 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.566941 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.567293 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.567480 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.567794 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.567900 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":130,"threads_started":1}
I20260928 02:45:20.568240 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.562074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.2.123.3:41589)
0928 02:45:20.563256 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.563268 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.563292 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:45:20.563367 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.563373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.563556 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.563710 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:20.563884 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.564645 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.565980 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.566814 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.567706 (+ 892us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.567943 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.567995 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.568089 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":135,"threads_started":1}
I20260928 02:45:20.570070 20246 tablet_service.cc:1467] Processing CreateTablet for tablet c0f27f527ac04a5aac4899825e4d29bf (DEFAULT_TABLE table=test-table [id=eeba0dd71fec4fec9b0b27d1a03a9c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:20.571118 20246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f27f527ac04a5aac4899825e4d29bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:20.571048 20508 tablet_service.cc:1467] Processing CreateTablet for tablet c0f27f527ac04a5aac4899825e4d29bf (DEFAULT_TABLE table=test-table [id=eeba0dd71fec4fec9b0b27d1a03a9c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:20.571918 20508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f27f527ac04a5aac4899825e4d29bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:20.572547 20377 tablet_service.cc:1467] Processing CreateTablet for tablet c0f27f527ac04a5aac4899825e4d29bf (DEFAULT_TABLE table=test-table [id=eeba0dd71fec4fec9b0b27d1a03a9c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:20.573508 20377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f27f527ac04a5aac4899825e4d29bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:20.577904 20591 tablet_bootstrap.cc:492] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: Bootstrap starting.
I20260928 02:45:20.579098 20592 tablet_bootstrap.cc:492] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1: Bootstrap starting.
I20260928 02:45:20.579892 20591 tablet_bootstrap.cc:654] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:20.580610 20591 log.cc:824] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:20.580826 20593 tablet_bootstrap.cc:492] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Bootstrap starting.
I20260928 02:45:20.581204 20592 tablet_bootstrap.cc:654] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:20.582043 20592 log.cc:824] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:20.582358 20591 tablet_bootstrap.cc:492] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: No bootstrap required, opened a new log
I20260928 02:45:20.582564 20591 ts_tablet_manager.cc:1397] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:45:20.582794 20593 tablet_bootstrap.cc:654] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:20.583534 20593 log.cc:824] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:20.584553 20592 tablet_bootstrap.cc:492] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1: No bootstrap required, opened a new log
I20260928 02:45:20.584740 20592 ts_tablet_manager.cc:1397] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:45:20.585475 20593 tablet_bootstrap.cc:492] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: No bootstrap required, opened a new log
I20260928 02:45:20.585654 20593 ts_tablet_manager.cc:1397] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:45:20.585938 20591 raft_consensus.cc:348] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.586215 20591 raft_consensus.cc:374] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:20.586277 20591 raft_consensus.cc:729] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31dda358d6464db5a3dd8b14011ed0, State: Initialized, Role: FOLLOWER
I20260928 02:45:20.586714 20591 consensus_queue.cc:260] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.587672 20591 ts_tablet_manager.cc:1428] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260928 02:45:20.588003 20574 heartbeater.cc:503] Master 127.2.123.62:43813 was elected leader, sending a full tablet report...
I20260928 02:45:20.588825 20592 raft_consensus.cc:348] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.589069 20592 raft_consensus.cc:374] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:20.589126 20592 raft_consensus.cc:729] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8786bd4d7d914574ba439b8b16a1a8a1, State: Initialized, Role: FOLLOWER
I20260928 02:45:20.589581 20592 consensus_queue.cc:260] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.589656 20593 raft_consensus.cc:348] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.589978 20593 raft_consensus.cc:374] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:20.590045 20593 raft_consensus.cc:729] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7e7d30c88e415faef6d5001a9f5647, State: Initialized, Role: FOLLOWER
I20260928 02:45:20.590389 20312 heartbeater.cc:503] Master 127.2.123.62:43813 was elected leader, sending a full tablet report...
I20260928 02:45:20.590595 20592 ts_tablet_manager.cc:1428] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:45:20.591421 20593 consensus_queue.cc:260] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.592060 20443 heartbeater.cc:503] Master 127.2.123.62:43813 was elected leader, sending a full tablet report...
I20260928 02:45:20.593221 20593 ts_tablet_manager.cc:1428] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260928 02:45:20.596107 20444 tablet.cc:2369] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:20.617730 20599 raft_consensus.cc:482] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:20.617923 20599 raft_consensus.cc:504] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.619057 20599 leader_election.cc:288] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8786bd4d7d914574ba439b8b16a1a8a1 (127.2.123.1:38875), bb31dda358d6464db5a3dd8b14011ed0 (127.2.123.3:41589)
I20260928 02:45:20.622481 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.619267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:38875 (local address 127.2.123.2:47851)
0928 02:45:20.619444 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.619459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:20.619554 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:20.619847 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:20.619850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:20.619868 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:20.619971 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.619977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.620688 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.620691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.621464 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.621473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.622112 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.622116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.622208 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.622234 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:45:20.622301 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:45:20.622377 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 02:45:20.622522 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.619356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:47851 (local address 127.2.123.1:38875)
0928 02:45:20.619463 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.619468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.619506 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:45:20.619682 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.619685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.619734 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.619812 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:20.620085 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.620589 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.621581 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.621995 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.622106 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.622169 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.622323 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.622397 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:45:20.622727 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.619314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:41589 (local address 127.2.123.2:38063)
0928 02:45:20.619710 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.619721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:20.619805 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:20.620021 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:20.620023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:20.620037 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:20.620142 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.620146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.620833 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.620836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.621589 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:20.621597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.622332 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.622335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:20.622427 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.622450 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:45:20.622523 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:45:20.622597 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":151,"threads_started":1}
I20260928 02:45:20.622864 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.619404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:38063 (local address 127.2.123.3:41589)
0928 02:45:20.619542 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.619546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.619797 (+ 251us) server_negotiation.cc:407] Connection header received
0928 02:45:20.619887 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.619890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.619930 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.619987 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:20.620242 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.620729 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.621708 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.622152 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.622245 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.622301 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.622552 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.622632 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:45:20.623481 20266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f27f527ac04a5aac4899825e4d29bf" candidate_uuid: "0e7e7d30c88e415faef6d5001a9f5647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" is_pre_election: true
I20260928 02:45:20.623481 20528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f27f527ac04a5aac4899825e4d29bf" candidate_uuid: "0e7e7d30c88e415faef6d5001a9f5647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31dda358d6464db5a3dd8b14011ed0" is_pre_election: true
I20260928 02:45:20.623848 20528 raft_consensus.cc:2454] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e7e7d30c88e415faef6d5001a9f5647 in term 0.
I20260928 02:45:20.623847 20266 raft_consensus.cc:2454] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e7e7d30c88e415faef6d5001a9f5647 in term 0.
I20260928 02:45:20.624395 20331 leader_election.cc:302] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7e7d30c88e415faef6d5001a9f5647, 8786bd4d7d914574ba439b8b16a1a8a1; no voters:
I20260928 02:45:20.624696 20599 raft_consensus.cc:2781] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:45:20.624781 20599 raft_consensus.cc:482] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:45:20.624825 20599 raft_consensus.cc:3037] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:20.625984 20599 raft_consensus.cc:504] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.626334 20599 leader_election.cc:288] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [CANDIDATE]: Term 1 election: Requested vote from peers 8786bd4d7d914574ba439b8b16a1a8a1 (127.2.123.1:38875), bb31dda358d6464db5a3dd8b14011ed0 (127.2.123.3:41589)
I20260928 02:45:20.626677 20528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f27f527ac04a5aac4899825e4d29bf" candidate_uuid: "0e7e7d30c88e415faef6d5001a9f5647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31dda358d6464db5a3dd8b14011ed0"
I20260928 02:45:20.626825 20528 raft_consensus.cc:3037] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:20.628125 20528 raft_consensus.cc:2454] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e7e7d30c88e415faef6d5001a9f5647 in term 1.
I20260928 02:45:20.628404 20266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f27f527ac04a5aac4899825e4d29bf" candidate_uuid: "0e7e7d30c88e415faef6d5001a9f5647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8786bd4d7d914574ba439b8b16a1a8a1"
I20260928 02:45:20.628463 20332 leader_election.cc:302] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7e7d30c88e415faef6d5001a9f5647, bb31dda358d6464db5a3dd8b14011ed0; no voters:
I20260928 02:45:20.628542 20266 raft_consensus.cc:3037] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:20.628620 20599 raft_consensus.cc:2781] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:20.628875 20599 raft_consensus.cc:686] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 1 LEADER]: Becoming Leader. State: Replica: 0e7e7d30c88e415faef6d5001a9f5647, State: Running, Role: LEADER
I20260928 02:45:20.629235 20599 consensus_queue.cc:237] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } }
I20260928 02:45:20.629724 20266 raft_consensus.cc:2454] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e7e7d30c88e415faef6d5001a9f5647 in term 1.
I20260928 02:45:20.632470 20123 catalog_manager.cc:5495] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e7e7d30c88e415faef6d5001a9f5647 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "0e7e7d30c88e415faef6d5001a9f5647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e7e7d30c88e415faef6d5001a9f5647" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 46639 } health_report { overall_health: HEALTHY } } }
I20260928 02:45:20.650113 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.646638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.2.123.2:46639)
0928 02:45:20.646807 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:20.646811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:20.646826 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:45:20.646904 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:20.646907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:20.646959 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:20.647043 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:20.647284 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.647791 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.648708 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:20.648927 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:20.649013 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:20.649303 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:20.649710 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:20.649829 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:20.649885 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:45:20.649936 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:45:20.655973 20266 raft_consensus.cc:1261] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1 [term 1 FOLLOWER]: Refusing update from remote peer 0e7e7d30c88e415faef6d5001a9f5647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:20.655973 20528 raft_consensus.cc:1261] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 1 FOLLOWER]: Refusing update from remote peer 0e7e7d30c88e415faef6d5001a9f5647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:20.656603 20599 consensus_queue.cc:1035] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8786bd4d7d914574ba439b8b16a1a8a1" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 38875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:20.656813 20601 consensus_queue.cc:1035] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31dda358d6464db5a3dd8b14011ed0" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:45:20.657398 20313 tablet.cc:2369] T c0f27f527ac04a5aac4899825e4d29bf P 8786bd4d7d914574ba439b8b16a1a8a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:20.667486 20606 mvcc.cc:205] Tried to move back new op lower bound from 7334148180598788096 to 7334148180499771392. Current Snapshot: MvccSnapshot[applied={T|T < 7334148180598788096}]
I20260928 02:45:20.673430 20123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:20.673593 20123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:20.675927 20123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f9a1ba77cc54bf685a4ecfa060340b0: Sending DeleteTablet for 3 replicas of tablet c0f27f527ac04a5aac4899825e4d29bf
I20260928 02:45:20.676682 20508 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f27f527ac04a5aac4899825e4d29bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:20 UTC) from {username='slave'} at 127.0.0.1:51296
I20260928 02:45:20.676774 20246 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f27f527ac04a5aac4899825e4d29bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:20 UTC) from {username='slave'} at 127.0.0.1:39364
I20260928 02:45:20.676827 20377 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f27f527ac04a5aac4899825e4d29bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:20 UTC) from {username='slave'} at 127.0.0.1:44364
I20260928 02:45:20.677313 20616 tablet_replica.cc:330] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: stopping tablet replica
I20260928 02:45:20.677620 20616 raft_consensus.cc:2229] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:45:20.677893 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 20184
I20260928 02:45:20.678124 20615 tablet_replica.cc:330] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: stopping tablet replica
I20260928 02:45:20.678519 20615 raft_consensus.cc:2229] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:20.678071 20616 raft_consensus.cc:2258] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:20.678818 20615 raft_consensus.cc:2258] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:20.680164 20616 ts_tablet_manager.cc:1905] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:20.681845 20615 ts_tablet_manager.cc:1905] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:20.682857 20616 ts_tablet_manager.cc:1918] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:20.682950 20616 log.cc:1196] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/c0f27f527ac04a5aac4899825e4d29bf
I20260928 02:45:20.683226 20616 ts_tablet_manager.cc:1939] T c0f27f527ac04a5aac4899825e4d29bf P 0e7e7d30c88e415faef6d5001a9f5647: Deleting consensus metadata
I20260928 02:45:20.684721 20615 ts_tablet_manager.cc:1918] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:20.684819 20615 log.cc:1196] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/c0f27f527ac04a5aac4899825e4d29bf
I20260928 02:45:20.685101 20615 ts_tablet_manager.cc:1939] T c0f27f527ac04a5aac4899825e4d29bf P bb31dda358d6464db5a3dd8b14011ed0: Deleting consensus metadata
W20260928 02:45:20.685593 20109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:38875 (error 108)
I20260928 02:45:20.685999 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 20315
I20260928 02:45:20.686165 20109 catalog_manager.cc:4841] TS 0e7e7d30c88e415faef6d5001a9f5647 (127.2.123.2:46639): tablet c0f27f527ac04a5aac4899825e4d29bf (table test-table [id=eeba0dd71fec4fec9b0b27d1a03a9c3e]) successfully deleted
I20260928 02:45:20.687518 20110 catalog_manager.cc:4841] TS bb31dda358d6464db5a3dd8b14011ed0 (127.2.123.3:41589): tablet c0f27f527ac04a5aac4899825e4d29bf (table test-table [id=eeba0dd71fec4fec9b0b27d1a03a9c3e]) successfully deleted
I20260928 02:45:20.688380 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:20.688083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:38875 (local address 127.0.0.1:39366)
0928 02:45:20.688233 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:20.688306 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:38875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:45:20.688531 20109 catalog_manager.cc:4584] TS 8786bd4d7d914574ba439b8b16a1a8a1 (127.2.123.1:38875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0f27f527ac04a5aac4899825e4d29bf: Network error: Client connection negotiation failed: client connection to 127.2.123.1:38875: connect: Connection refused (error 111)
I20260928 02:45:20.691160 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 20446
I20260928 02:45:20.695852 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 20093
2026-09-28T02:45:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:20 dist-test-slave-rqvb krb5kdc[20620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:20 dist-test-slave-rqvb krb5kdc[20620](info): set up 2 sockets
Sep 28 02:45:20 dist-test-slave-rqvb krb5kdc[20620](info): commencing operation
krb5kdc: starting...
W20260928 02:45:21.649842 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:21 dist-test-slave-rqvb krb5kdc[20620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:21.770542 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:42275
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:42275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:21.883481 20633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:21.883766 20633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:21.883827 20633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:21.887863 20633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:21.887920 20633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:21.887946 20633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:21.887969 20633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:21.887997 20633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:21.893276 20633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:42275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:42275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:21.894416 20633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:21.895424 20633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:21.903818 20638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:21.903831 20639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:21.903831 20641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:21.903874 20633 server_base.cc:1034] running on GCE node
I20260928 02:45:21.904642 20633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:21.905869 20633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:21.907048 20633 hybrid_clock.cc:648] HybridClock initialized: now 1790563521907025 us; error 35 us; skew 500 ppm
Sep 28 02:45:21 dist-test-slave-rqvb krb5kdc[20620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:21.914362 20633 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:21.915278 20633 webserver.cc:460] Webserver started at http://127.2.123.62:34195/ using document root <none> and password file <none>
I20260928 02:45:21.915807 20633 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:21.915864 20633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:21.916081 20633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:21.918164 20633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "47a1ef75c03e464194a64729e4d40dae"
format_stamp: "Formatted at 2026-09-28 02:45:21 on dist-test-slave-rqvb"
server_key: "8e8bbdbe3cd367cffe92a75404801f60"
server_key_iv: "274993ad2f62b3af1443a907ed36e7e6"
server_key_version: "encryptionkey@0"
I20260928 02:45:21.918697 20633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "47a1ef75c03e464194a64729e4d40dae"
format_stamp: "Formatted at 2026-09-28 02:45:21 on dist-test-slave-rqvb"
server_key: "8e8bbdbe3cd367cffe92a75404801f60"
server_key_iv: "274993ad2f62b3af1443a907ed36e7e6"
server_key_version: "encryptionkey@0"
I20260928 02:45:21.922564 20633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:45:21.925052 20648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:21.926384 20633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:21.926498 20633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "47a1ef75c03e464194a64729e4d40dae"
format_stamp: "Formatted at 2026-09-28 02:45:21 on dist-test-slave-rqvb"
server_key: "8e8bbdbe3cd367cffe92a75404801f60"
server_key_iv: "274993ad2f62b3af1443a907ed36e7e6"
server_key_version: "encryptionkey@0"
I20260928 02:45:21.926616 20633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:21.942319 20633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:21.945624 20633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:21.945822 20633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:21.952880 20633 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:42275
I20260928 02:45:21.952909 20700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:42275 every 8 connection(s)
I20260928 02:45:21.953863 20633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:21.956733 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 20633
I20260928 02:45:21.956775 20701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:21.956851 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:21.957129 2540 external_mini_cluster.cc:1428] Setting key a4a1979416f94de5d4b88d7e2eaa354a
I20260928 02:45:21.964732 20701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae: Bootstrap starting.
Sep 28 02:45:21 dist-test-slave-rqvb krb5kdc[20620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:21.967512 20701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:21.968243 20701 log.cc:824] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:21.970273 20701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae: No bootstrap required, opened a new log
I20260928 02:45:21.971650 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:21.958609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.2.123.62:42275)
0928 02:45:21.959027 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:21.959042 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:21.959084 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:45:21.959804 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:21.959824 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:21.960161 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:21.960468 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:21.961416 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:21.962528 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:21.963709 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:21.964191 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:21.964312 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:21.967266 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:21.967272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:21.967285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:21.967315 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:21.968936 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:21.969357 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:21.969362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:21.969368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:21.969480 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:21.969740 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:21.969744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:21.969745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:21.970187 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:21.970365 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:21.970649 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:45:21.970943 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260928 02:45:21.973938 20701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1ef75c03e464194a64729e4d40dae" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42275 } }
I20260928 02:45:21.974200 20701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:21.974267 20701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a1ef75c03e464194a64729e4d40dae, State: Initialized, Role: FOLLOWER
I20260928 02:45:21.974666 20701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1ef75c03e464194a64729e4d40dae" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42275 } }
I20260928 02:45:21.974802 20701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:21.974864 20701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:21.974961 20701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:21.975847 20701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1ef75c03e464194a64729e4d40dae" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42275 } }
I20260928 02:45:21.976171 20701 leader_election.cc:302] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47a1ef75c03e464194a64729e4d40dae; no voters:
I20260928 02:45:21.976508 20701 leader_election.cc:288] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:21.976627 20706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:21.976826 20706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [term 1 LEADER]: Becoming Leader. State: Replica: 47a1ef75c03e464194a64729e4d40dae, State: Running, Role: LEADER
I20260928 02:45:21.977102 20706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1ef75c03e464194a64729e4d40dae" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42275 } }
I20260928 02:45:21.977743 20701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:21.978755 20708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a1ef75c03e464194a64729e4d40dae. Latest consensus state: current_term: 1 leader_uuid: "47a1ef75c03e464194a64729e4d40dae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1ef75c03e464194a64729e4d40dae" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42275 } } }
I20260928 02:45:21.978857 20708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:21.978724 20707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47a1ef75c03e464194a64729e4d40dae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a1ef75c03e464194a64729e4d40dae" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 42275 } } }
I20260928 02:45:21.979107 20707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:21.979256 20715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:21.982535 20715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:21.988062 20715 catalog_manager.cc:1347] Generated new cluster ID: f857c8bee3cc4a82a450b588b98cc9ea
I20260928 02:45:21.988126 20715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:22.007493 20715 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:22.008657 20715 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:22.015928 20715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae: Generated new TSK 0
I20260928 02:45:22.016659 20715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:45:22.046466 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42275
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:22.155540 20727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:22.155804 20727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:22.155848 20727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:22.159865 20727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:22.159929 20727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:22.159991 20727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:45:22.165274 20727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:22.166399 20727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:22.167445 20727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:22.177282 20735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:22.177373 20727 server_base.cc:1034] running on GCE node
W20260928 02:45:22.177321 20733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:22.177285 20732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:22.177881 20727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:22.178588 20727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:22.179742 20727 hybrid_clock.cc:648] HybridClock initialized: now 1790563522179713 us; error 42 us; skew 500 ppm
Sep 28 02:45:22 dist-test-slave-rqvb krb5kdc[20620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:22.182521 20727 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:45:22.183377 20727 webserver.cc:460] Webserver started at http://127.2.123.1:39577/ using document root <none> and password file <none>
I20260928 02:45:22.183887 20727 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:22.183943 20727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:22.184100 20727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:22.186097 20727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "7db8001d5b704daf9273e9004eb0e47e"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "d5108dc31b9bf5468e7f93ee6bee3a6f"
server_key_iv: "75ffea708636667b07201b2ef27132b2"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.186615 20727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "7db8001d5b704daf9273e9004eb0e47e"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "d5108dc31b9bf5468e7f93ee6bee3a6f"
server_key_iv: "75ffea708636667b07201b2ef27132b2"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.190145 20727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:45:22.192497 20742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.193636 20727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:22.193727 20727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "7db8001d5b704daf9273e9004eb0e47e"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "d5108dc31b9bf5468e7f93ee6bee3a6f"
server_key_iv: "75ffea708636667b07201b2ef27132b2"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.193809 20727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:22.205295 20727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:22.208504 20727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:22.208658 20727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:22.209192 20727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:22.209954 20727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:22.210062 20727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.210101 20727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:22.210121 20727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.219098 20727 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:39947
I20260928 02:45:22.219177 20855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:39947 every 8 connection(s)
I20260928 02:45:22.220017 20727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:45:22.222328 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 20727
I20260928 02:45:22.222465 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:45:22.222797 2540 external_mini_cluster.cc:1428] Setting key ff3aa7e931b1df6ca455b9c441c41045
Sep 28 02:45:22 dist-test-slave-rqvb krb5kdc[20620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
I20260928 02:45:22.234217 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.221864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:58893 (local address 127.2.123.62:42275)
0928 02:45:22.222029 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.222033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.223100 (+ 1067us) server_negotiation.cc:407] Connection header received
0928 02:45:22.224225 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.224228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.224275 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.224353 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:22.225493 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.226239 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.227367 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.227605 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.227692 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.230445 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:22.230449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:22.230452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:22.230484 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:22.232190 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:22.232684 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:22.232688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:22.232689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:22.232772 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:22.233059 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:22.233061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:22.233062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:22.233238 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:22.233310 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.233930 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.234079 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:45:22.235111 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.222403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42275 (local address 127.2.123.1:58893)
0928 02:45:22.222963 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.223007 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:22.223997 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:22.224510 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:22.224517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:22.224871 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:22.225346 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.225360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.226360 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.226363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.227247 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.227257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.227743 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.227747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.227907 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.228513 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:22.228529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:22.230265 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:22.232313 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:22.232318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:22.232332 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:22.232587 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:22.232842 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:22.232844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:22.232846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:22.232984 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:22.233333 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:22.233338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:22.233655 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 02:45:22.233887 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 02:45:22.234285 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":185,"threads_started":1}
I20260928 02:45:22.236898 20856 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42275
I20260928 02:45:22.237133 20856 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:22.237627 20856 heartbeater.cc:511] Master 127.2.123.62:42275 requested a full tablet report, sending...
I20260928 02:45:22.239274 20665 ts_manager.cc:194] Registered new tserver with Master: 7db8001d5b704daf9273e9004eb0e47e (127.2.123.1:39947)
I20260928 02:45:22.240391 20665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:58893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:45:22.249148 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42275
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:22.355682 20861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:22.355942 20861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:22.355997 20861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:22.359984 20861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:22.360046 20861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:22.360123 20861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:45:22.365248 20861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:22.366333 20861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:22.367390 20861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:22.376581 20866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:22.376590 20867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:22.376600 20869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:22.376782 20861 server_base.cc:1034] running on GCE node
I20260928 02:45:22.377261 20861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:22.377890 20861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:22.379055 20861 hybrid_clock.cc:648] HybridClock initialized: now 1790563522379018 us; error 49 us; skew 500 ppm
Sep 28 02:45:22 dist-test-slave-rqvb krb5kdc[20620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:22.381873 20861 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:45:22.382712 20861 webserver.cc:460] Webserver started at http://127.2.123.2:42667/ using document root <none> and password file <none>
I20260928 02:45:22.383255 20861 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:22.383324 20861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:22.383538 20861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:22.385565 20861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "5ffdba4bb8852c43cede24c97f43654f"
server_key_iv: "c7b842e713619d1f29cbf072d512880a"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.386132 20861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "5ffdba4bb8852c43cede24c97f43654f"
server_key_iv: "c7b842e713619d1f29cbf072d512880a"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.389678 20861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:22.392170 20876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.393424 20861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:45:22.393530 20861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "5ffdba4bb8852c43cede24c97f43654f"
server_key_iv: "c7b842e713619d1f29cbf072d512880a"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.393625 20861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:22.406090 20861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:22.409193 20861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:22.409374 20861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:22.409945 20861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:22.410809 20861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:22.410866 20861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.410918 20861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:22.410944 20861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.419297 20861 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:43131
I20260928 02:45:22.419337 20989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:43131 every 8 connection(s)
I20260928 02:45:22.420159 20861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:45:22.425196 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 20861
I20260928 02:45:22.425325 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:45:22.425583 2540 external_mini_cluster.cc:1428] Setting key 75d7906192af0669e4f40ee355694f65
Sep 28 02:45:22 dist-test-slave-rqvb krb5kdc[20620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:22.433416 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.421847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:53283 (local address 127.2.123.62:42275)
0928 02:45:22.422034 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.422039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.422910 (+ 871us) server_negotiation.cc:407] Connection header received
0928 02:45:22.423735 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.423737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.423857 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.423933 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:22.425061 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.425684 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.427061 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.427289 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.427371 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.429871 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:22.429874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:22.429876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:22.429899 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:22.431301 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:22.431849 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:22.431852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:22.431853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:22.431925 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:22.432315 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:22.432317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:22.432319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:22.432500 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:22.432585 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.433176 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.433281 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:45:22.434543 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.422314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42275 (local address 127.2.123.2:53283)
0928 02:45:22.422754 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.422797 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:22.423547 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:22.424084 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:22.424091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:22.424457 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:22.424917 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.424930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.425808 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.425812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.426919 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.426928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.427444 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.427448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.427632 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.428182 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:22.428202 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:22.429720 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:22.431402 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:22.431408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:22.431422 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:22.431753 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:22.432005 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:22.432009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:22.432010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:22.432186 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:22.432600 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:22.432607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:22.432914 (+ 307us) client_negotiation.cc:769] Sending connection context
0928 02:45:22.433136 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 02:45:22.433469 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:45:22.435711 20990 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42275
I20260928 02:45:22.435943 20990 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:22.436398 20990 heartbeater.cc:511] Master 127.2.123.62:42275 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:45:22.437496 20665 ts_manager.cc:194] Registered new tserver with Master: 1f1d070f39bb4cf2b1781a8e7d6dd136 (127.2.123.2:43131)
I20260928 02:45:22.438323 20665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:53283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:45:22.453240 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42275
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:22.561821 20995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:22.562126 20995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:22.562170 20995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:22.566179 20995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:22.566246 20995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:22.566308 20995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:45:22.571583 20995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:42275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:22.572629 20995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:22.573678 20995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:22.583118 20995 server_base.cc:1034] running on GCE node
W20260928 02:45:22.583062 21000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:22.583079 21003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:22.583050 21001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:22.583631 20995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:22.584237 20995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:22.585404 20995 hybrid_clock.cc:648] HybridClock initialized: now 1790563522585379 us; error 42 us; skew 500 ppm
Sep 28 02:45:22 dist-test-slave-rqvb krb5kdc[20620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:22.588037 20995 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:45:22.588871 20995 webserver.cc:460] Webserver started at http://127.2.123.3:34373/ using document root <none> and password file <none>
I20260928 02:45:22.589387 20995 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:22.589449 20995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:22.589639 20995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:22.591621 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "46ea5d80d4d34d5dab895a07eef15e4b"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "f0fbc3e735067cf472c2ab9354a396f7"
server_key_iv: "bb445c818cc29ade8abf0b70fcae724d"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.592132 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "46ea5d80d4d34d5dab895a07eef15e4b"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "f0fbc3e735067cf472c2ab9354a396f7"
server_key_iv: "bb445c818cc29ade8abf0b70fcae724d"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.595621 20995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:45:22.597941 21010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.599172 20995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:22.599272 20995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "46ea5d80d4d34d5dab895a07eef15e4b"
format_stamp: "Formatted at 2026-09-28 02:45:22 on dist-test-slave-rqvb"
server_key: "f0fbc3e735067cf472c2ab9354a396f7"
server_key_iv: "bb445c818cc29ade8abf0b70fcae724d"
server_key_version: "encryptionkey@0"
I20260928 02:45:22.599371 20995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:22.624176 20995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:22.627249 20995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:22.627434 20995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:22.627992 20995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:22.628784 20995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:22.628831 20995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.628885 20995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:22.628911 20995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:22.637413 20995 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:40937
I20260928 02:45:22.637492 21123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:40937 every 8 connection(s)
I20260928 02:45:22.638326 20995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:45:22.639350 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 20995
I20260928 02:45:22.639464 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:45:22.639751 2540 external_mini_cluster.cc:1428] Setting key dad1e9cd1f2c56de58e881b97e89bcdd
Sep 28 02:45:22 dist-test-slave-rqvb krb5kdc[20620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:22.651396 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.640229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:41249 (local address 127.2.123.62:42275)
0928 02:45:22.640351 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.640354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.641386 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 02:45:22.642331 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.642335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.642389 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.642460 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:22.643595 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.644126 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.645227 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.645456 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.645542 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.647977 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:22.647980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:22.647982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:22.648008 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:22.649311 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:22.649796 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:22.649799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:22.649800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:22.649958 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:22.650351 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:22.650354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:22.650355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:22.650566 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:22.650660 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.651154 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.651268 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:45:22.652227 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.640707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:42275 (local address 127.2.123.3:41249)
0928 02:45:22.641254 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.641297 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:22.642144 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:22.642616 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:22.642625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:22.642983 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:22.643464 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.643477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.644237 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.644239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.645113 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.645122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.645586 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.645590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.645750 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.646326 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:22.646341 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:22.647750 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:22.649418 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:22.649423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:22.649434 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:22.649683 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:22.650099 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:22.650101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:22.650103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:22.650240 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:22.650658 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:22.650662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:22.650901 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 02:45:22.651086 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 02:45:22.651344 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":182,"threads_started":1}
I20260928 02:45:22.653368 21124 heartbeater.cc:347] Connected to a master server at 127.2.123.62:42275
I20260928 02:45:22.653622 21124 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:22.654156 21124 heartbeater.cc:511] Master 127.2.123.62:42275 requested a full tablet report, sending...
I20260928 02:45:22.655180 20665 ts_manager.cc:194] Registered new tserver with Master: 46ea5d80d4d34d5dab895a07eef15e4b (127.2.123.3:40937)
I20260928 02:45:22.655733 20665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:41249
I20260928 02:45:22.665503 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:45:22.675225 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.667478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.2.123.62:42275)
0928 02:45:22.667635 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.667639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.667739 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:45:22.667838 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.667840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.667882 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.667937 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:22.668548 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.669066 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.669914 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.670202 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.670287 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.671812 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:22.671815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:22.671817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:22.671841 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:22.673238 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:22.673726 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:22.673729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:22.673731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:22.673808 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:22.674217 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:22.674219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:22.674220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:22.674437 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:22.674550 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.674925 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.675037 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:45:22.678550 20665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:45:22.681790 20665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:45:22.696306 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.690296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:39947 (local address 127.0.0.1:35158)
0928 02:45:22.690659 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.690711 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:22.690838 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:22.692220 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:22.692223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:22.692258 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:22.692373 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.692380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.693867 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.693870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.694985 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.694995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.695757 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.695760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.695879 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.695920 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:45:22.696016 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:45:22.696157 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":67,"thread_start_us":87,"threads_started":1}
I20260928 02:45:22.696346 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.691011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.2.123.1:39947)
0928 02:45:22.691614 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.691628 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.691650 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:45:22.691721 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.691726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.691934 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.692532 (+ 598us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:22.692675 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.693752 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.695093 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.695638 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.695855 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.696001 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.696049 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.696141 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":128,"threads_started":1}
I20260928 02:45:22.696507 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.691126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.2.123.3:40937)
0928 02:45:22.691817 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.691830 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.691849 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:45:22.691912 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.691917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.692061 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.692220 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:22.692477 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.693639 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.695140 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.695897 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.695998 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.696127 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.696223 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.696303 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":114,"threads_started":1}
I20260928 02:45:22.696992 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.690348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:43131 (local address 127.0.0.1:50094)
0928 02:45:22.691143 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.691156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:22.691244 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:22.692015 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:22.692021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:22.692045 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:22.692195 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.692202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.693347 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.693349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.694638 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.694649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.695720 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.695723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.696722 (+ 999us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.696741 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:45:22.696804 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:45:22.696870 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":29,"thread_start_us":101,"threads_started":1}
I20260928 02:45:22.697085 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.691115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.2.123.2:43131)
0928 02:45:22.691530 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.691542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.691564 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:45:22.691635 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.691639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.691831 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.692346 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:22.692491 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.693218 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.694773 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.695596 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.695703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.695850 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.696856 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.696949 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":113,"threads_started":1}
I20260928 02:45:22.697315 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.690643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:40937 (local address 127.0.0.1:47262)
0928 02:45:22.691423 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.691435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:22.691519 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:22.692195 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:22.692198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:22.692213 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:22.692366 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.692372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.693773 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.693777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.695016 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.695026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.696004 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.696008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.696099 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.696122 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:45:22.696196 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:45:22.696267 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":87,"spinlock_wait_cycles":68864,"thread_start_us":90,"threads_started":1}
I20260928 02:45:22.699841 20790 tablet_service.cc:1467] Processing CreateTablet for tablet a17de1a267db4c04aefb39831ef8b023 (DEFAULT_TABLE table=test-table [id=07e3a10a87ab4190bb236e6c0f16be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:22.699841 20924 tablet_service.cc:1467] Processing CreateTablet for tablet a17de1a267db4c04aefb39831ef8b023 (DEFAULT_TABLE table=test-table [id=07e3a10a87ab4190bb236e6c0f16be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:22.700987 20924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17de1a267db4c04aefb39831ef8b023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:22.700984 20790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17de1a267db4c04aefb39831ef8b023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:22.703950 21058 tablet_service.cc:1467] Processing CreateTablet for tablet a17de1a267db4c04aefb39831ef8b023 (DEFAULT_TABLE table=test-table [id=07e3a10a87ab4190bb236e6c0f16be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:22.704876 21058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17de1a267db4c04aefb39831ef8b023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:22.708437 21141 tablet_bootstrap.cc:492] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e: Bootstrap starting.
I20260928 02:45:22.708963 21142 tablet_bootstrap.cc:492] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: Bootstrap starting.
I20260928 02:45:22.711458 21141 tablet_bootstrap.cc:654] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:22.711671 21142 tablet_bootstrap.cc:654] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:22.711834 21143 tablet_bootstrap.cc:492] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: Bootstrap starting.
I20260928 02:45:22.712559 21142 log.cc:824] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:22.713315 21141 log.cc:824] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:22.714010 21143 tablet_bootstrap.cc:654] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:22.714849 21142 tablet_bootstrap.cc:492] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: No bootstrap required, opened a new log
I20260928 02:45:22.714989 21143 log.cc:824] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:22.715067 21142 ts_tablet_manager.cc:1397] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:45:22.715477 21141 tablet_bootstrap.cc:492] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e: No bootstrap required, opened a new log
I20260928 02:45:22.715667 21141 ts_tablet_manager.cc:1397] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260928 02:45:22.716665 21143 tablet_bootstrap.cc:492] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: No bootstrap required, opened a new log
I20260928 02:45:22.716835 21143 ts_tablet_manager.cc:1397] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:45:22.719094 21142 raft_consensus.cc:348] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.719427 21142 raft_consensus.cc:374] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:22.719480 21142 raft_consensus.cc:729] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1d070f39bb4cf2b1781a8e7d6dd136, State: Initialized, Role: FOLLOWER
I20260928 02:45:22.719461 21141 raft_consensus.cc:348] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.719679 21141 raft_consensus.cc:374] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:22.719722 21141 raft_consensus.cc:729] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db8001d5b704daf9273e9004eb0e47e, State: Initialized, Role: FOLLOWER
I20260928 02:45:22.719672 21143 raft_consensus.cc:348] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.720013 21143 raft_consensus.cc:374] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:22.719966 21142 consensus_queue.cc:260] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.720067 21143 raft_consensus.cc:729] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ea5d80d4d34d5dab895a07eef15e4b, State: Initialized, Role: FOLLOWER
I20260928 02:45:22.720165 21141 consensus_queue.cc:260] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.720481 21143 consensus_queue.cc:260] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.721040 21142 ts_tablet_manager.cc:1428] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:45:22.721040 21141 ts_tablet_manager.cc:1428] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 02:45:22.721401 20856 heartbeater.cc:503] Master 127.2.123.62:42275 was elected leader, sending a full tablet report...
I20260928 02:45:22.721402 20990 heartbeater.cc:503] Master 127.2.123.62:42275 was elected leader, sending a full tablet report...
I20260928 02:45:22.721838 21124 heartbeater.cc:503] Master 127.2.123.62:42275 was elected leader, sending a full tablet report...
I20260928 02:45:22.721854 21143 ts_tablet_manager.cc:1428] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:45:22.766839 21149 raft_consensus.cc:482] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:22.767181 21149 raft_consensus.cc:504] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.768435 21149 leader_election.cc:288] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db8001d5b704daf9273e9004eb0e47e (127.2.123.1:39947), 1f1d070f39bb4cf2b1781a8e7d6dd136 (127.2.123.2:43131)
I20260928 02:45:22.772109 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.768607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:39947 (local address 127.2.123.3:52783)
0928 02:45:22.768775 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.768789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:22.768872 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:22.769187 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:22.769191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:22.769210 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:22.769332 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.769339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.770249 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.770253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.771035 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.771044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.771688 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.771691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.771791 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.771812 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:45:22.771875 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:45:22.771937 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 02:45:22.772107 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.768768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:52783 (local address 127.2.123.1:39947)
0928 02:45:22.768917 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.768922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.768938 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:45:22.768991 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.768994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.769040 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.769154 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:22.769457 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.770149 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.771162 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.771578 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.771656 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.771706 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.771902 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.771976 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:45:22.772145 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.769024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:52873 (local address 127.2.123.2:43131)
0928 02:45:22.769145 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.769149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.769163 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:45:22.769213 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.769215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.769255 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.769329 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:22.769666 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.770211 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.771258 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.771661 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.771758 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.771817 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.771971 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.772044 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:45:22.772147 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.768667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:43131 (local address 127.2.123.3:52873)
0928 02:45:22.769054 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.769066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:22.769148 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:22.769374 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:22.769376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:22.769396 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:22.769539 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.769546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.770339 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.770342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.771141 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:22.771152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.771772 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.771775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:22.771853 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.771876 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:45:22.771949 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:45:22.772008 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":121,"threads_started":1}
I20260928 02:45:22.772769 20810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17de1a267db4c04aefb39831ef8b023" candidate_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db8001d5b704daf9273e9004eb0e47e" is_pre_election: true
I20260928 02:45:22.772761 20944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17de1a267db4c04aefb39831ef8b023" candidate_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" is_pre_election: true
I20260928 02:45:22.773036 20810 raft_consensus.cc:2454] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46ea5d80d4d34d5dab895a07eef15e4b in term 0.
I20260928 02:45:22.773051 20944 raft_consensus.cc:2454] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46ea5d80d4d34d5dab895a07eef15e4b in term 0.
I20260928 02:45:22.773535 21014 leader_election.cc:302] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1d070f39bb4cf2b1781a8e7d6dd136, 46ea5d80d4d34d5dab895a07eef15e4b; no voters:
I20260928 02:45:22.773837 21149 raft_consensus.cc:2781] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:45:22.773924 21149 raft_consensus.cc:482] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:45:22.773980 21149 raft_consensus.cc:3037] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:22.774963 21149 raft_consensus.cc:504] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.775300 21149 leader_election.cc:288] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [CANDIDATE]: Term 1 election: Requested vote from peers 7db8001d5b704daf9273e9004eb0e47e (127.2.123.1:39947), 1f1d070f39bb4cf2b1781a8e7d6dd136 (127.2.123.2:43131)
I20260928 02:45:22.775668 20944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17de1a267db4c04aefb39831ef8b023" candidate_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136"
I20260928 02:45:22.775810 20944 raft_consensus.cc:3037] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:22.776852 20944 raft_consensus.cc:2454] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46ea5d80d4d34d5dab895a07eef15e4b in term 1.
I20260928 02:45:22.777127 21014 leader_election.cc:302] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1d070f39bb4cf2b1781a8e7d6dd136, 46ea5d80d4d34d5dab895a07eef15e4b; no voters:
I20260928 02:45:22.777120 20810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17de1a267db4c04aefb39831ef8b023" candidate_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db8001d5b704daf9273e9004eb0e47e"
I20260928 02:45:22.777285 21149 raft_consensus.cc:2781] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:22.777282 20810 raft_consensus.cc:3037] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:22.777508 21149 raft_consensus.cc:686] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 1 LEADER]: Becoming Leader. State: Replica: 46ea5d80d4d34d5dab895a07eef15e4b, State: Running, Role: LEADER
I20260928 02:45:22.777837 21149 consensus_queue.cc:237] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } }
I20260928 02:45:22.778693 20810 raft_consensus.cc:2454] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46ea5d80d4d34d5dab895a07eef15e4b in term 1.
I20260928 02:45:22.781019 20663 catalog_manager.cc:5495] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 46ea5d80d4d34d5dab895a07eef15e4b (127.2.123.3). New cstate: current_term: 1 leader_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46ea5d80d4d34d5dab895a07eef15e4b" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 40937 } health_report { overall_health: HEALTHY } } }
I20260928 02:45:22.796267 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.792749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.2.123.3:40937)
0928 02:45:22.792868 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:22.792873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:22.792889 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:45:22.793077 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:22.793079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:22.793133 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:22.793218 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:22.793511 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.794043 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.794862 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:22.795077 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:22.795160 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:22.795467 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:22.795858 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:22.795998 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:22.796083 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:45:22.796119 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:45:22.802930 20810 raft_consensus.cc:1261] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [term 1 FOLLOWER]: Refusing update from remote peer 46ea5d80d4d34d5dab895a07eef15e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:22.802942 20944 raft_consensus.cc:1261] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 1 FOLLOWER]: Refusing update from remote peer 46ea5d80d4d34d5dab895a07eef15e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:22.803661 21151 consensus_queue.cc:1035] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f1d070f39bb4cf2b1781a8e7d6dd136" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:22.803871 21149 consensus_queue.cc:1035] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db8001d5b704daf9273e9004eb0e47e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:22.812948 21156 mvcc.cc:205] Tried to move back new op lower bound from 7334148189392814080 to 7334148189300711424. Current Snapshot: MvccSnapshot[applied={T|T < 7334148189392814080}]
I20260928 02:45:22.821602 20664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:22.821823 20664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:22.824515 20664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47a1ef75c03e464194a64729e4d40dae: Sending DeleteTablet for 3 replicas of tablet a17de1a267db4c04aefb39831ef8b023
I20260928 02:45:22.825322 20790 tablet_service.cc:1514] Processing DeleteTablet for tablet a17de1a267db4c04aefb39831ef8b023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:22 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:35158
I20260928 02:45:22.825330 20924 tablet_service.cc:1514] Processing DeleteTablet for tablet a17de1a267db4c04aefb39831ef8b023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:22 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:50094
I20260928 02:45:22.825889 21166 tablet_replica.cc:330] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e: stopping tablet replica
I20260928 02:45:22.826014 21058 tablet_service.cc:1514] Processing DeleteTablet for tablet a17de1a267db4c04aefb39831ef8b023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:22 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:47262
I20260928 02:45:22.826421 21166 raft_consensus.cc:2229] T a17de1a267db4c04aefb39831ef8b023 P 7db8001d5b704daf9273e9004eb0e47e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:22.826651 21165 tablet_replica.cc:330] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: stopping tablet replica
I20260928 02:45:22.826719 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 20727
I20260928 02:45:22.826942 21165 raft_consensus.cc:2229] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:22.827790 21165 raft_consensus.cc:2258] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:22.828670 21167 tablet_replica.cc:330] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: stopping tablet replica
I20260928 02:45:22.828930 21167 raft_consensus.cc:2229] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:45:22.829332 21167 raft_consensus.cc:2258] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:22.829751 21165 ts_tablet_manager.cc:1905] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:22.830772 21167 ts_tablet_manager.cc:1905] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:22.833079 21165 ts_tablet_manager.cc:1918] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:22.833256 21165 log.cc:1196] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/a17de1a267db4c04aefb39831ef8b023
I20260928 02:45:22.833376 21167 ts_tablet_manager.cc:1918] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:22.833473 21167 log.cc:1196] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/a17de1a267db4c04aefb39831ef8b023
I20260928 02:45:22.833767 21167 ts_tablet_manager.cc:1939] T a17de1a267db4c04aefb39831ef8b023 P 46ea5d80d4d34d5dab895a07eef15e4b: Deleting consensus metadata
I20260928 02:45:22.833786 21165 ts_tablet_manager.cc:1939] T a17de1a267db4c04aefb39831ef8b023 P 1f1d070f39bb4cf2b1781a8e7d6dd136: Deleting consensus metadata
W20260928 02:45:22.834292 20651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:39947 (error 108)
I20260928 02:45:22.835031 20649 catalog_manager.cc:4841] TS 46ea5d80d4d34d5dab895a07eef15e4b (127.2.123.3:40937): tablet a17de1a267db4c04aefb39831ef8b023 (table test-table [id=07e3a10a87ab4190bb236e6c0f16be98]) successfully deleted
I20260928 02:45:22.835104 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 20861
I20260928 02:45:22.835165 20652 catalog_manager.cc:4841] TS 1f1d070f39bb4cf2b1781a8e7d6dd136 (127.2.123.2:43131): tablet a17de1a267db4c04aefb39831ef8b023 (table test-table [id=07e3a10a87ab4190bb236e6c0f16be98]) successfully deleted
I20260928 02:45:22.838321 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:22.837956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:39947 (local address 127.0.0.1:35172)
0928 02:45:22.838127 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:22.838223 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:39947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:45:22.838587 20651 catalog_manager.cc:4584] TS 7db8001d5b704daf9273e9004eb0e47e (127.2.123.1:39947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a17de1a267db4c04aefb39831ef8b023: Network error: Client connection negotiation failed: client connection to 127.2.123.1:39947: connect: Connection refused (error 111)
I20260928 02:45:22.840577 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 20995
I20260928 02:45:22.845417 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 20633
2026-09-28T02:45:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2149 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:22Z Disabled control of system clock
I20260928 02:45:22.873517 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33597
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:45:22.985474 21172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:22.985782 21172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:22.985819 21172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:22.989940 21172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:22.990036 21172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:22.990051 21172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:22.990067 21172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:22.990082 21172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:22.995193 21172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:33597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:22.996291 21172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:22.997649 21172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:23.006247 21178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:23.006477 21172 server_base.cc:1034] running on GCE node
W20260928 02:45:23.006248 21177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:23.006248 21180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:23.007449 21172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:23.008579 21172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:23.009763 21172 hybrid_clock.cc:648] HybridClock initialized: now 1790563523009730 us; error 46 us; skew 500 ppm
I20260928 02:45:23.011782 21172 webserver.cc:460] Webserver started at http://127.2.123.62:37433/ using document root <none> and password file <none>
I20260928 02:45:23.012334 21172 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:23.012398 21172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:23.012612 21172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:23.014722 21172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "44d42379e5164c79976879dade5dd0f4"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "061a79c2f8c40ffdc8dcf80df231ff11"
server_key_iv: "bfc8490974b43038da9d114dc5b00602"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.015252 21172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "44d42379e5164c79976879dade5dd0f4"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "061a79c2f8c40ffdc8dcf80df231ff11"
server_key_iv: "bfc8490974b43038da9d114dc5b00602"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.018953 21172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:45:23.021396 21186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.022776 21172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:23.022903 21172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "44d42379e5164c79976879dade5dd0f4"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "061a79c2f8c40ffdc8dcf80df231ff11"
server_key_iv: "bfc8490974b43038da9d114dc5b00602"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.023007 21172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:23.041182 21172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:23.041918 21172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:23.042163 21172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:23.048853 21172 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:33597
I20260928 02:45:23.048888 21238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:33597 every 8 connection(s)
I20260928 02:45:23.049830 21172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:23.053074 21239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:23.058665 21239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4: Bootstrap starting.
I20260928 02:45:23.059336 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 21172
I20260928 02:45:23.059453 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:23.059739 2540 external_mini_cluster.cc:1428] Setting key 2c3053e8d2ee25d7e2f6d227d81bd53b
I20260928 02:45:23.061439 21239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:23.062209 21239 log.cc:824] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:23.064064 21239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4: No bootstrap required, opened a new log
I20260928 02:45:23.067744 21239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d42379e5164c79976879dade5dd0f4" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33597 } }
I20260928 02:45:23.068024 21239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:23.068096 21239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d42379e5164c79976879dade5dd0f4, State: Initialized, Role: FOLLOWER
I20260928 02:45:23.068572 21239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d42379e5164c79976879dade5dd0f4" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33597 } }
I20260928 02:45:23.068696 21239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:23.068758 21239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:23.068851 21239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:23.069015 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.060902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.2.123.62:33597)
0928 02:45:23.061554 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.061573 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.061617 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:45:23.062299 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.062321 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.062757 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.063136 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:23.063556 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.064563 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.065451 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.065943 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.066142 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.067039 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:23.067045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:23.067059 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:23.067508 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:23.067589 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:23.067612 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:23.067699 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.067847 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.068145 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260928 02:45:23.069808 21239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d42379e5164c79976879dade5dd0f4" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33597 } }
I20260928 02:45:23.070182 21239 leader_election.cc:302] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44d42379e5164c79976879dade5dd0f4; no voters:
I20260928 02:45:23.070514 21239 leader_election.cc:288] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:23.070632 21244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:23.070847 21244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 44d42379e5164c79976879dade5dd0f4, State: Running, Role: LEADER
I20260928 02:45:23.071234 21244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d42379e5164c79976879dade5dd0f4" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33597 } }
I20260928 02:45:23.071782 21239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:23.072784 21245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44d42379e5164c79976879dade5dd0f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d42379e5164c79976879dade5dd0f4" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33597 } } }
I20260928 02:45:23.072907 21245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:23.073153 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44d42379e5164c79976879dade5dd0f4. Latest consensus state: current_term: 1 leader_uuid: "44d42379e5164c79976879dade5dd0f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d42379e5164c79976879dade5dd0f4" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33597 } } }
I20260928 02:45:23.073253 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:23.073345 21253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:23.076200 21253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:23.081632 21253 catalog_manager.cc:1347] Generated new cluster ID: 8fc7ff9b7ad0474abea815813f3aef20
I20260928 02:45:23.081712 21253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:23.102558 21253 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:23.103637 21253 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:23.110463 21253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4: Generated new TSK 0
I20260928 02:45:23.111317 21253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:23.117487 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33597
--builtin_ntp_servers=127.2.123.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:45:23.231233 21263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:23.231518 21263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:23.231557 21263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:23.235883 21263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:23.235953 21263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:23.236016 21263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:45:23.241024 21263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:23.242337 21263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:23.243772 21263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:23.253187 21269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:23.253187 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:23.253337 21263 server_base.cc:1034] running on GCE node
W20260928 02:45:23.253521 21271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:23.253913 21263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:23.254578 21263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:23.255731 21263 hybrid_clock.cc:648] HybridClock initialized: now 1790563523255704 us; error 42 us; skew 500 ppm
I20260928 02:45:23.257584 21263 webserver.cc:460] Webserver started at http://127.2.123.1:35989/ using document root <none> and password file <none>
I20260928 02:45:23.258181 21263 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:23.258250 21263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:23.258450 21263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:23.260496 21263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "443c95c89c74468babe5b87117fd2928"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "cda980130db519b2b1e640ff8940d7eb"
server_key_iv: "d22763b8cc471296244a9ce347675abc"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.261040 21263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "443c95c89c74468babe5b87117fd2928"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "cda980130db519b2b1e640ff8940d7eb"
server_key_iv: "d22763b8cc471296244a9ce347675abc"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.264793 21263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:45:23.267378 21277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.268762 21263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:23.268882 21263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "443c95c89c74468babe5b87117fd2928"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "cda980130db519b2b1e640ff8940d7eb"
server_key_iv: "d22763b8cc471296244a9ce347675abc"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.268981 21263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:23.293268 21263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:23.294195 21263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:23.294610 21263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:23.295295 21263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:23.296311 21263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:23.296362 21263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.296407 21263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:23.296422 21263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.304844 21263 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:43741
I20260928 02:45:23.304885 21390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:43741 every 8 connection(s)
I20260928 02:45:23.305745 21263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:45:23.314146 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 21263
I20260928 02:45:23.314265 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:45:23.314571 2540 external_mini_cluster.cc:1428] Setting key e783aa39279f33989bcc6ad5a36afdc1
I20260928 02:45:23.315207 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.307701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:45023 (local address 127.2.123.62:33597)
0928 02:45:23.307867 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.307871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.308961 (+ 1090us) server_negotiation.cc:407] Connection header received
0928 02:45:23.310437 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.310440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.310501 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.310570 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:23.311567 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.312107 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.313247 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.313497 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.313586 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.314730 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:23.314733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:23.314735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:23.314749 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:23.314768 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:23.314777 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:23.314812 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.315000 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.315062 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:45:23.316111 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.308099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33597 (local address 127.2.123.1:45023)
0928 02:45:23.308796 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.308838 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:23.310206 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:23.310762 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:23.310770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:23.310863 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:23.311418 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.311436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.312232 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.312235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.313126 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.313136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.313647 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.313651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.313806 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.314442 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:23.314443 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:23.314486 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:23.314552 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:23.314863 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:23.314870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:23.314874 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:45:23.314975 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:45:23.315259 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":119,"threads_started":1}
I20260928 02:45:23.316504 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33597
--builtin_ntp_servers=127.2.123.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:45:23.317390 21391 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33597
I20260928 02:45:23.317610 21391 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:23.318197 21391 heartbeater.cc:511] Master 127.2.123.62:33597 requested a full tablet report, sending...
I20260928 02:45:23.319614 21200 ts_manager.cc:194] Registered new tserver with Master: 443c95c89c74468babe5b87117fd2928 (127.2.123.1:43741)
I20260928 02:45:23.320740 21200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.1:45023
W20260928 02:45:23.424612 21394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:23.424882 21394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:23.424935 21394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:23.429001 21394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:23.429060 21394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:23.429134 21394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:45:23.433923 21394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:23.434962 21394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:23.436271 21394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:23.445698 21402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:23.445791 21394 server_base.cc:1034] running on GCE node
W20260928 02:45:23.445698 21400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:23.445708 21399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:23.446395 21394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:23.447036 21394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:23.448210 21394 hybrid_clock.cc:648] HybridClock initialized: now 1790563523448180 us; error 38 us; skew 500 ppm
I20260928 02:45:23.450122 21394 webserver.cc:460] Webserver started at http://127.2.123.2:43211/ using document root <none> and password file <none>
I20260928 02:45:23.450662 21394 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:23.450730 21394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:23.450928 21394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:23.453028 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "28b266f9c591482a8ac0b826a957dd6f"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "64597d926f09a5e6d0a02b88505b9d63"
server_key_iv: "f6b0e92fc3b656befed9253f5064bf2f"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.453567 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "28b266f9c591482a8ac0b826a957dd6f"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "64597d926f09a5e6d0a02b88505b9d63"
server_key_iv: "f6b0e92fc3b656befed9253f5064bf2f"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.457202 21394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 02:45:23.459635 21408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.460852 21394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:23.460956 21394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "28b266f9c591482a8ac0b826a957dd6f"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "64597d926f09a5e6d0a02b88505b9d63"
server_key_iv: "f6b0e92fc3b656befed9253f5064bf2f"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.461097 21394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:23.482740 21394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:23.483441 21394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:23.483624 21394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:23.484215 21394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:23.485180 21394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:23.485227 21394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.485280 21394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:23.485307 21394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.493678 21394 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:36841
I20260928 02:45:23.493742 21521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:36841 every 8 connection(s)
I20260928 02:45:23.494588 21394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:45:23.502894 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 21394
I20260928 02:45:23.503005 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:45:23.503315 2540 external_mini_cluster.cc:1428] Setting key 4e7357b845238fccfa8a01a27a71b749
I20260928 02:45:23.503749 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.496427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:50213 (local address 127.2.123.62:33597)
0928 02:45:23.496602 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.496607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.497643 (+ 1036us) server_negotiation.cc:407] Connection header received
0928 02:45:23.499046 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.499048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.499096 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.499169 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:23.500124 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.500657 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.501834 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.502120 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.502201 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.503256 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:23.503260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:23.503262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:23.503277 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:23.503299 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:23.503309 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:23.503384 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.503547 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.503609 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:45:23.504786 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.496831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33597 (local address 127.2.123.2:50213)
0928 02:45:23.497500 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.497541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:23.498844 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:23.499377 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:23.499384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:23.499474 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:23.499986 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.500000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.500771 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.500773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.501717 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.501727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.502245 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.502248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.502386 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.502971 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:23.502973 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:23.503021 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:23.503099 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:23.503396 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:23.503406 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:23.503412 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:45:23.503532 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:45:23.503781 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260928 02:45:23.505403 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33597
--builtin_ntp_servers=127.2.123.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:45:23.505919 21522 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33597
I20260928 02:45:23.506191 21522 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:23.506799 21522 heartbeater.cc:511] Master 127.2.123.62:33597 requested a full tablet report, sending...
I20260928 02:45:23.507984 21200 ts_manager.cc:194] Registered new tserver with Master: 28b266f9c591482a8ac0b826a957dd6f (127.2.123.2:36841)
I20260928 02:45:23.508574 21200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.2:50213
W20260928 02:45:23.614934 21525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:23.615209 21525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:23.615264 21525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:23.619272 21525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:23.619330 21525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:23.619405 21525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:45:23.624245 21525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:23.625365 21525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:23.626845 21525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:23.636346 21525 server_base.cc:1034] running on GCE node
W20260928 02:45:23.636288 21531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:23.636282 21530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:23.636353 21533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:23.637089 21525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:23.637742 21525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:23.638885 21525 hybrid_clock.cc:648] HybridClock initialized: now 1790563523638852 us; error 46 us; skew 500 ppm
I20260928 02:45:23.640708 21525 webserver.cc:460] Webserver started at http://127.2.123.3:37029/ using document root <none> and password file <none>
I20260928 02:45:23.641260 21525 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:23.641319 21525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:23.641506 21525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:23.643600 21525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "710b4e80bed54b3c94a1274052b20030"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "826c54279061161e287419f5d96c2d65"
server_key_iv: "ed7bb42d749a6a1ac53d63cb37145b0d"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.644122 21525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "710b4e80bed54b3c94a1274052b20030"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "826c54279061161e287419f5d96c2d65"
server_key_iv: "ed7bb42d749a6a1ac53d63cb37145b0d"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.647800 21525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:45:23.650158 21539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.651315 21525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:23.651407 21525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "710b4e80bed54b3c94a1274052b20030"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-rqvb"
server_key: "826c54279061161e287419f5d96c2d65"
server_key_iv: "ed7bb42d749a6a1ac53d63cb37145b0d"
server_key_version: "encryptionkey@0"
I20260928 02:45:23.651494 21525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:23.663623 21525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:23.664332 21525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:23.664501 21525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:23.665064 21525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:23.666085 21525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:23.666132 21525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.666174 21525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:23.666203 21525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.674855 21525 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:33419
I20260928 02:45:23.674901 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:33419 every 8 connection(s)
I20260928 02:45:23.675777 21525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:45:23.681882 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 21525
I20260928 02:45:23.682022 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:45:23.682284 2540 external_mini_cluster.cc:1428] Setting key a8467e0dba4b3c34025e33dff346074f
I20260928 02:45:23.685323 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.677522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:59235 (local address 127.2.123.62:33597)
0928 02:45:23.677651 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.677654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.678817 (+ 1163us) server_negotiation.cc:407] Connection header received
0928 02:45:23.680179 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.680182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.680240 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.680315 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:23.681266 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.681856 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.683264 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.683501 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.683653 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.684791 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:23.684794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:23.684796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:23.684807 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:23.684826 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:23.684834 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:23.684892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.685111 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.685185 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:45:23.686275 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.677936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33597 (local address 127.2.123.3:59235)
0928 02:45:23.678669 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.678711 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:23.679980 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:23.680495 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:23.680503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:23.680603 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:23.681126 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.681140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.682033 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.682037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.683125 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.683136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.683708 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.683712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.683910 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.684529 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:23.684531 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:23.684572 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:23.684634 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:23.684963 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:23.684973 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:23.684979 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:45:23.685097 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 02:45:23.685377 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260928 02:45:23.687330 21653 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33597
I20260928 02:45:23.687537 21653 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:23.688015 21653 heartbeater.cc:511] Master 127.2.123.62:33597 requested a full tablet report, sending...
I20260928 02:45:23.688951 21200 ts_manager.cc:194] Registered new tserver with Master: 710b4e80bed54b3c94a1274052b20030 (127.2.123.3:33419)
I20260928 02:45:23.689603 21200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.3:59235
I20260928 02:45:23.696072 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:45:23.701551 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.697996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.2.123.62:33597)
0928 02:45:23.698110 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.698114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.698161 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:45:23.698264 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.698266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.698301 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.698363 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:23.698600 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.699153 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.699938 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.700196 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.700275 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.701131 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:23.701134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:23.701137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:23.701152 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:23.701173 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:23.701182 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:23.701292 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.701411 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.701444 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:45:23.704262 21200 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:45:23.707296 21200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:45:23.720783 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.715231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:43741 (local address 127.0.0.1:59814)
0928 02:45:23.715651 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.715715 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:23.715835 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:23.717252 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:23.717255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:23.717271 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:23.717346 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.717351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.718373 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.718377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.719491 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.719500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.720248 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.720251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.720353 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.720393 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:45:23.720503 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:45:23.720632 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":66,"threads_started":1}
I20260928 02:45:23.720805 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.716195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.2.123.1:43741)
0928 02:45:23.716830 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.716840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.716858 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:45:23.716920 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.716924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.717089 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.717263 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:23.717444 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.718251 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.719608 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.720150 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.720254 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.720446 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.720528 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.720643 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":146,"threads_started":1}
I20260928 02:45:23.721433 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.715909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:33419 (local address 127.0.0.1:33786)
0928 02:45:23.716708 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.716720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:23.716809 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:23.717377 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:23.717379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:23.717397 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:23.717495 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.717501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.718864 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.718867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.720036 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.720047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.721064 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.721068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.721155 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.721177 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:45:23.721233 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:45:23.721302 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260928 02:45:23.721534 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.716331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.2.123.3:33419)
0928 02:45:23.716671 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.716680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.716815 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:45:23.716899 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.716905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.717095 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.717237 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:23.717616 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.718756 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.720172 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.720956 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.721053 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.721246 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.721295 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.721385 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":143,"threads_started":1}
I20260928 02:45:23.722353 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.715355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:36841 (local address 127.0.0.1:57486)
0928 02:45:23.716276 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.716289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:23.716378 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:23.717054 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:23.717061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:23.717083 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:23.717219 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.717227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.718694 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.718697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.720093 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.720101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.721954 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.721980 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.722080 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.722098 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:45:23.722160 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:45:23.722229 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":221,"thread_start_us":114,"threads_started":1}
I20260928 02:45:23.722461 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.715575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.2.123.2:36841)
0928 02:45:23.716139 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.716153 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.716475 (+ 322us) server_negotiation.cc:407] Connection header received
0928 02:45:23.716624 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.716629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.716850 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.717067 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:23.717345 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.718558 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.720212 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.721377 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.721702 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.721894 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.722209 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.722317 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":122,"threads_started":1}
I20260928 02:45:23.724257 21325 tablet_service.cc:1467] Processing CreateTablet for tablet 76fdb904c3e844038b409b686b6e7c7e (DEFAULT_TABLE table=test-table [id=0510598138584d45aee9c7161865a9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:23.724515 21456 tablet_service.cc:1467] Processing CreateTablet for tablet 76fdb904c3e844038b409b686b6e7c7e (DEFAULT_TABLE table=test-table [id=0510598138584d45aee9c7161865a9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:23.724494 21587 tablet_service.cc:1467] Processing CreateTablet for tablet 76fdb904c3e844038b409b686b6e7c7e (DEFAULT_TABLE table=test-table [id=0510598138584d45aee9c7161865a9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:23.725342 21456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fdb904c3e844038b409b686b6e7c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:23.725368 21325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fdb904c3e844038b409b686b6e7c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:23.725366 21587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fdb904c3e844038b409b686b6e7c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:23.731585 21670 tablet_bootstrap.cc:492] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Bootstrap starting.
I20260928 02:45:23.733148 21672 tablet_bootstrap.cc:492] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928: Bootstrap starting.
I20260928 02:45:23.733913 21670 tablet_bootstrap.cc:654] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:23.734275 21671 tablet_bootstrap.cc:492] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: Bootstrap starting.
I20260928 02:45:23.734809 21670 log.cc:824] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:23.735236 21672 tablet_bootstrap.cc:654] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:23.736028 21672 log.cc:824] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:23.736090 21671 tablet_bootstrap.cc:654] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:23.736759 21671 log.cc:824] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:23.736802 21670 tablet_bootstrap.cc:492] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: No bootstrap required, opened a new log
I20260928 02:45:23.736959 21670 ts_tablet_manager.cc:1397] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 02:45:23.738515 21671 tablet_bootstrap.cc:492] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: No bootstrap required, opened a new log
I20260928 02:45:23.738692 21671 ts_tablet_manager.cc:1397] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:45:23.739589 21672 tablet_bootstrap.cc:492] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928: No bootstrap required, opened a new log
I20260928 02:45:23.739765 21672 ts_tablet_manager.cc:1397] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 02:45:23.740872 21670 raft_consensus.cc:348] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
I20260928 02:45:23.741137 21670 raft_consensus.cc:374] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:23.741185 21670 raft_consensus.cc:729] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b266f9c591482a8ac0b826a957dd6f, State: Initialized, Role: FOLLOWER
I20260928 02:45:23.741667 21670 consensus_queue.cc:260] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
I20260928 02:45:23.742022 21671 raft_consensus.cc:348] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
I20260928 02:45:23.742264 21671 raft_consensus.cc:374] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:23.742318 21671 raft_consensus.cc:729] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710b4e80bed54b3c94a1274052b20030, State: Initialized, Role: FOLLOWER
I20260928 02:45:23.742336 21522 heartbeater.cc:503] Master 127.2.123.62:33597 was elected leader, sending a full tablet report...
I20260928 02:45:23.742671 21670 ts_tablet_manager.cc:1428] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 02:45:23.742704 21671 consensus_queue.cc:260] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
I20260928 02:45:23.743539 21653 heartbeater.cc:503] Master 127.2.123.62:33597 was elected leader, sending a full tablet report...
I20260928 02:45:23.743587 21671 ts_tablet_manager.cc:1428] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:45:23.744374 21672 raft_consensus.cc:348] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
I20260928 02:45:23.744635 21672 raft_consensus.cc:374] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:23.744689 21672 raft_consensus.cc:729] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c95c89c74468babe5b87117fd2928, State: Initialized, Role: FOLLOWER
I20260928 02:45:23.745007 21672 consensus_queue.cc:260] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
W20260928 02:45:23.745517 21523 tablet.cc:2369] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:23.745898 21672 ts_tablet_manager.cc:1428] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:45:23.746253 21391 heartbeater.cc:503] Master 127.2.123.62:33597 was elected leader, sending a full tablet report...
I20260928 02:45:23.754930 21677 raft_consensus.cc:482] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:23.755128 21677 raft_consensus.cc:504] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
I20260928 02:45:23.756134 21677 leader_election.cc:288] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c95c89c74468babe5b87117fd2928 (127.2.123.1:43741), 28b266f9c591482a8ac0b826a957dd6f (127.2.123.2:36841)
I20260928 02:45:23.759712 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.756314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:43741 (local address 127.2.123.3:38165)
0928 02:45:23.756486 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.756501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:23.756582 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:23.756925 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:23.756928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:23.756942 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:23.757050 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.757057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.757781 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.757784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.758705 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.758714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.759317 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.759320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.759414 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.759432 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:45:23.759497 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:45:23.759566 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":20}
I20260928 02:45:23.759711 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.756448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:38165 (local address 127.2.123.1:43741)
0928 02:45:23.756562 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.756566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.756579 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:45:23.756701 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.756705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.756752 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.756820 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:23.757177 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.757676 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.758825 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.759205 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.759309 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.759367 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.759528 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.759596 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:45:23.759750 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.756707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:37933 (local address 127.2.123.2:36841)
0928 02:45:23.756817 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.756821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.756837 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:45:23.756916 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.756918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.756964 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.757036 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:23.757290 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.757785 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.758877 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.759258 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.759346 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.759402 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.759565 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.759638 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:45:23.759771 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.756344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:36841 (local address 127.2.123.3:37933)
0928 02:45:23.756755 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.756767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:23.756836 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:23.757059 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:23.757062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:23.757080 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:23.757190 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.757194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.757881 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.757885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.758761 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:23.758771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.759357 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.759360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:23.759455 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.759477 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:45:23.759542 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:45:23.759614 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260928 02:45:23.760272 21476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fdb904c3e844038b409b686b6e7c7e" candidate_uuid: "710b4e80bed54b3c94a1274052b20030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b266f9c591482a8ac0b826a957dd6f" is_pre_election: true
I20260928 02:45:23.760346 21345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fdb904c3e844038b409b686b6e7c7e" candidate_uuid: "710b4e80bed54b3c94a1274052b20030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c95c89c74468babe5b87117fd2928" is_pre_election: true
I20260928 02:45:23.760528 21476 raft_consensus.cc:2454] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710b4e80bed54b3c94a1274052b20030 in term 0.
I20260928 02:45:23.760582 21345 raft_consensus.cc:2454] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 710b4e80bed54b3c94a1274052b20030 in term 0.
I20260928 02:45:23.761068 21542 leader_election.cc:302] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28b266f9c591482a8ac0b826a957dd6f, 443c95c89c74468babe5b87117fd2928, 710b4e80bed54b3c94a1274052b20030; no voters:
I20260928 02:45:23.761313 21677 raft_consensus.cc:2781] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:45:23.761385 21677 raft_consensus.cc:482] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:45:23.761415 21677 raft_consensus.cc:3037] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:23.762408 21677 raft_consensus.cc:504] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
I20260928 02:45:23.762746 21677 leader_election.cc:288] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [CANDIDATE]: Term 1 election: Requested vote from peers 443c95c89c74468babe5b87117fd2928 (127.2.123.1:43741), 28b266f9c591482a8ac0b826a957dd6f (127.2.123.2:36841)
I20260928 02:45:23.763059 21476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fdb904c3e844038b409b686b6e7c7e" candidate_uuid: "710b4e80bed54b3c94a1274052b20030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b266f9c591482a8ac0b826a957dd6f"
I20260928 02:45:23.763056 21345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fdb904c3e844038b409b686b6e7c7e" candidate_uuid: "710b4e80bed54b3c94a1274052b20030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c95c89c74468babe5b87117fd2928"
I20260928 02:45:23.763160 21345 raft_consensus.cc:3037] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:23.763165 21476 raft_consensus.cc:3037] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:23.764153 21476 raft_consensus.cc:2454] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710b4e80bed54b3c94a1274052b20030 in term 1.
I20260928 02:45:23.764164 21345 raft_consensus.cc:2454] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710b4e80bed54b3c94a1274052b20030 in term 1.
I20260928 02:45:23.764429 21540 leader_election.cc:302] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b266f9c591482a8ac0b826a957dd6f, 710b4e80bed54b3c94a1274052b20030; no voters:
I20260928 02:45:23.764572 21677 raft_consensus.cc:2781] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:23.764804 21677 raft_consensus.cc:686] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 1 LEADER]: Becoming Leader. State: Replica: 710b4e80bed54b3c94a1274052b20030, State: Running, Role: LEADER
I20260928 02:45:23.765144 21677 consensus_queue.cc:237] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } }
I20260928 02:45:23.768098 21200 catalog_manager.cc:5495] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 reported cstate change: term changed from 0 to 1, leader changed from <none> to 710b4e80bed54b3c94a1274052b20030 (127.2.123.3). New cstate: current_term: 1 leader_uuid: "710b4e80bed54b3c94a1274052b20030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710b4e80bed54b3c94a1274052b20030" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33419 } health_report { overall_health: HEALTHY } } }
I20260928 02:45:23.788368 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.784857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.2.123.3:33419)
0928 02:45:23.785034 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.785038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.785065 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:45:23.785132 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.785134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.785191 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.785277 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:23.785534 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.786105 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.786948 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.787157 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.787241 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.787549 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:23.787955 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:23.788084 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.788192 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.788227 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:45:23.794111 21345 raft_consensus.cc:1261] T 76fdb904c3e844038b409b686b6e7c7e P 443c95c89c74468babe5b87117fd2928 [term 1 FOLLOWER]: Refusing update from remote peer 710b4e80bed54b3c94a1274052b20030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:23.794142 21476 raft_consensus.cc:1261] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 1 FOLLOWER]: Refusing update from remote peer 710b4e80bed54b3c94a1274052b20030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:23.794799 21677 consensus_queue.cc:1035] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c95c89c74468babe5b87117fd2928" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:23.794974 21680 consensus_queue.cc:1035] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b266f9c591482a8ac0b826a957dd6f" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:23.800637 21683 mvcc.cc:205] Tried to move back new op lower bound from 7334148193453740032 to 7334148193344548864. Current Snapshot: MvccSnapshot[applied={T|T < 7334148193453740032}]
I20260928 02:45:23.801254 21685 mvcc.cc:205] Tried to move back new op lower bound from 7334148193453740032 to 7334148193344548864. Current Snapshot: MvccSnapshot[applied={T|T < 7334148193453740032}]
I20260928 02:45:23.805070 21684 mvcc.cc:205] Tried to move back new op lower bound from 7334148193453740032 to 7334148193344548864. Current Snapshot: MvccSnapshot[applied={T|T < 7334148193453740032}]
I20260928 02:45:23.811240 21203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:23.811385 21203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:23.813630 21203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44d42379e5164c79976879dade5dd0f4: Sending DeleteTablet for 3 replicas of tablet 76fdb904c3e844038b409b686b6e7c7e
I20260928 02:45:23.814401 21456 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fdb904c3e844038b409b686b6e7c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:23 UTC) from {username='slave'} at 127.0.0.1:57486
I20260928 02:45:23.814544 21325 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fdb904c3e844038b409b686b6e7c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:23 UTC) from {username='slave'} at 127.0.0.1:59814
I20260928 02:45:23.814920 21692 tablet_replica.cc:330] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: stopping tablet replica
I20260928 02:45:23.814999 21587 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fdb904c3e844038b409b686b6e7c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:23 UTC) from {username='slave'} at 127.0.0.1:33786
I20260928 02:45:23.815295 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 21263
I20260928 02:45:23.815426 21692 raft_consensus.cc:2229] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:23.815790 21692 raft_consensus.cc:2258] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:23.817554 21692 ts_tablet_manager.cc:1905] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:23.819818 21692 ts_tablet_manager.cc:1918] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:23.819905 21692 log.cc:1196] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/76fdb904c3e844038b409b686b6e7c7e
I20260928 02:45:23.820163 21692 ts_tablet_manager.cc:1939] T 76fdb904c3e844038b409b686b6e7c7e P 28b266f9c591482a8ac0b826a957dd6f: Deleting consensus metadata
I20260928 02:45:23.821301 21187 catalog_manager.cc:4841] TS 28b266f9c591482a8ac0b826a957dd6f (127.2.123.2:36841): tablet 76fdb904c3e844038b409b686b6e7c7e (table test-table [id=0510598138584d45aee9c7161865a9d3]) successfully deleted
I20260928 02:45:23.822152 21694 tablet_replica.cc:330] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: stopping tablet replica
I20260928 02:45:23.822414 21694 raft_consensus.cc:2229] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:45:23.822769 21694 raft_consensus.cc:2258] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:45:23.823024 21189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:43741 (error 108)
I20260928 02:45:23.823145 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 21394
I20260928 02:45:23.824398 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.824119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:43741 (local address 127.0.0.1:59828)
0928 02:45:23.824241 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:23.824314 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:45:23.824719 21189 catalog_manager.cc:4584] TS 443c95c89c74468babe5b87117fd2928 (127.2.123.1:43741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76fdb904c3e844038b409b686b6e7c7e: Network error: Client connection negotiation failed: client connection to 127.2.123.1:43741: connect: Connection refused (error 111)
I20260928 02:45:23.824880 21694 ts_tablet_manager.cc:1905] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:23.827086 21694 ts_tablet_manager.cc:1918] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:23.827171 21694 log.cc:1196] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/76fdb904c3e844038b409b686b6e7c7e
I20260928 02:45:23.827409 21694 ts_tablet_manager.cc:1939] T 76fdb904c3e844038b409b686b6e7c7e P 710b4e80bed54b3c94a1274052b20030: Deleting consensus metadata
I20260928 02:45:23.828378 21189 catalog_manager.cc:4841] TS 710b4e80bed54b3c94a1274052b20030 (127.2.123.3:33419): tablet 76fdb904c3e844038b409b686b6e7c7e (table test-table [id=0510598138584d45aee9c7161865a9d3]) successfully deleted
I20260928 02:45:23.830214 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 21525
I20260928 02:45:23.834868 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 21172
2026-09-28T02:45:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:23 dist-test-slave-rqvb krb5kdc[21698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:23 dist-test-slave-rqvb krb5kdc[21698](info): set up 2 sockets
Sep 28 02:45:23 dist-test-slave-rqvb krb5kdc[21698](info): commencing operation
krb5kdc: starting...
W20260928 02:45:24.785671 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:24 dist-test-slave-rqvb krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:24.905278 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:39251
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:39251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:25.012380 21711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:25.012660 21711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:25.012698 21711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:25.016705 21711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:25.016769 21711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:25.016788 21711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:25.016803 21711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:25.016816 21711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:25.022040 21711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:39251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:39251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:25.023016 21711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:25.023974 21711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:25.032263 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:25.032294 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:25.032281 21711 server_base.cc:1034] running on GCE node
W20260928 02:45:25.032301 21719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:25.032979 21711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:25.034149 21711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:25.035313 21711 hybrid_clock.cc:648] HybridClock initialized: now 1790563525035296 us; error 29 us; skew 500 ppm
Sep 28 02:45:25 dist-test-slave-rqvb krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:25.037930 21711 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:25.038839 21711 webserver.cc:460] Webserver started at http://127.2.123.62:35941/ using document root <none> and password file <none>
I20260928 02:45:25.039387 21711 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:25.039453 21711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:25.039659 21711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:25.041740 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "f333d1fe9bd1422d7d277503a86b5e44"
server_key_iv: "4de150a77c17df24a9c787dc0ff1885b"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.042304 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "f333d1fe9bd1422d7d277503a86b5e44"
server_key_iv: "4de150a77c17df24a9c787dc0ff1885b"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.045849 21711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:25.048118 21726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.049392 21711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:25.049523 21711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "f333d1fe9bd1422d7d277503a86b5e44"
server_key_iv: "4de150a77c17df24a9c787dc0ff1885b"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.049626 21711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:25.071556 21711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:25.074705 21711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:25.074887 21711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:25.081871 21711 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:39251
I20260928 02:45:25.081939 21778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:39251 every 8 connection(s)
I20260928 02:45:25.082901 21711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:25.085850 21779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:25.091475 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 21711
I20260928 02:45:25.091624 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:25.091776 21779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36: Bootstrap starting.
I20260928 02:45:25.091876 2540 external_mini_cluster.cc:1428] Setting key d919fbd4b1fb6807570d5f298241746e
I20260928 02:45:25.094532 21779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:25.095441 21779 log.cc:824] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:25.097841 21779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36: No bootstrap required, opened a new log
I20260928 02:45:25.102181 21779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39251 } }
I20260928 02:45:25.102414 21779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:25.102470 21779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122ee37aa8eb48c8baa4c9dd9bd4da36, State: Initialized, Role: FOLLOWER
I20260928 02:45:25.102890 21779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39251 } }
Sep 28 02:45:25 dist-test-slave-rqvb krb5kdc[21698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:25.103017 21779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:25.103094 21779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:25.103196 21779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:25.104344 21779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39251 } }
I20260928 02:45:25.104740 21779 leader_election.cc:302] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 122ee37aa8eb48c8baa4c9dd9bd4da36; no voters:
I20260928 02:45:25.105197 21779 leader_election.cc:288] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:25.105340 21784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:25.105510 21784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [term 1 LEADER]: Becoming Leader. State: Replica: 122ee37aa8eb48c8baa4c9dd9bd4da36, State: Running, Role: LEADER
I20260928 02:45:25.105814 21784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39251 } }
I20260928 02:45:25.106621 21779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:25.107559 21786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 122ee37aa8eb48c8baa4c9dd9bd4da36. Latest consensus state: current_term: 1 leader_uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39251 } } }
I20260928 02:45:25.107699 21786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:25.107976 21785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122ee37aa8eb48c8baa4c9dd9bd4da36" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 39251 } } }
I20260928 02:45:25.108073 21785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:25.108371 21793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:25.109555 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.093244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.2.123.62:39251)
0928 02:45:25.093692 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.093710 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.093754 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:45:25.094619 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.094643 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.095087 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.095448 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:25.096397 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.097871 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.099127 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.100040 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.100208 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.103701 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:25.103708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:25.103725 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:25.103760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:25.105848 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.106351 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.106356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.106363 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.106491 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.106776 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.106780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.106782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.107301 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:25.107512 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:25.108266 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 02:45:25.108583 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260928 02:45:25.111672 21793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:25.117615 21793 catalog_manager.cc:1347] Generated new cluster ID: c93c922d0aab4ebdade10902b86187c2
I20260928 02:45:25.117717 21793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:25.130393 21793 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:25.131356 21793 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:25.145288 21793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36: Generated new TSK 0
I20260928 02:45:25.146116 21793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
Entry for principal kudu/127.2.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab.
I20260928 02:45:25.173486 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39251
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:25.283322 21805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:25.283587 21805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:25.283628 21805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:25.287722 21805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:25.287797 21805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:25.287855 21805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:45:25.293293 21805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.1.keytab
--principal=kudu/127.2.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:25.294405 21805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:25.295455 21805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:25.305154 21811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:25.305238 21805 server_base.cc:1034] running on GCE node
W20260928 02:45:25.305161 21810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:25.305250 21813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:25.305871 21805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:25.306581 21805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:25.307747 21805 hybrid_clock.cc:648] HybridClock initialized: now 1790563525307731 us; error 36 us; skew 500 ppm
Sep 28 02:45:25 dist-test-slave-rqvb krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:25.310745 21805 init.cc:377] Logged in from keytab as kudu/127.2.123.1@KRBTEST.COM (short username kudu)
I20260928 02:45:25.311630 21805 webserver.cc:460] Webserver started at http://127.2.123.1:35367/ using document root <none> and password file <none>
I20260928 02:45:25.312162 21805 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:25.312223 21805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:25.312397 21805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:25.314512 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "d1b76c42e05647928e5138aa4f9f3050"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "de33d94af01ca77c1f584b6d67ad03b3"
server_key_iv: "e01ed3d5fcb8727a4bbdec84dd21f54b"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.315034 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "d1b76c42e05647928e5138aa4f9f3050"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "de33d94af01ca77c1f584b6d67ad03b3"
server_key_iv: "e01ed3d5fcb8727a4bbdec84dd21f54b"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.318753 21805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:25.321347 21820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.322649 21805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:25.322751 21805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "d1b76c42e05647928e5138aa4f9f3050"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "de33d94af01ca77c1f584b6d67ad03b3"
server_key_iv: "e01ed3d5fcb8727a4bbdec84dd21f54b"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.322863 21805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:25.337918 21805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:25.341061 21805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:25.341235 21805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:25.341809 21805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:25.342644 21805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:25.342687 21805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.342728 21805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:25.342756 21805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.351083 21805 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:40877
I20260928 02:45:25.351114 21933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:40877 every 8 connection(s)
I20260928 02:45:25.351945 21805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:45:25.359754 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 21805
I20260928 02:45:25.359884 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:45:25.360152 2540 external_mini_cluster.cc:1428] Setting key f419f360da368d56357261474d872999
Sep 28 02:45:25 dist-test-slave-rqvb krb5kdc[21698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.1@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:25.365784 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.353828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:39531 (local address 127.2.123.62:39251)
0928 02:45:25.353982 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.353987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.354870 (+ 883us) server_negotiation.cc:407] Connection header received
0928 02:45:25.356122 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.356125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.356179 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.356248 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:25.357393 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.357919 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.359038 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.359268 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.359353 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.361921 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:25.361924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:25.361926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:25.361950 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:25.363574 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.364119 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.364122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.364124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.364251 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.364593 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.364596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.364604 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.364820 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:25.364913 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:25.365405 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 02:45:25.365559 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:45:25.366709 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.354296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39251 (local address 127.2.123.1:39531)
0928 02:45:25.354729 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.354772 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.355921 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:25.356416 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:25.356423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:25.356781 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:25.357254 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.357268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.358054 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.358056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.358924 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.358933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.359403 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.359407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.359559 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.360095 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:25.360111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:25.361747 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:25.363746 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:25.363750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:25.363763 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:25.364022 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:25.364343 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:25.364345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:25.364346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:25.364479 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:25.364932 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:25.364937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:25.365175 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 02:45:25.365336 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 02:45:25.365676 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":116,"threads_started":1}
I20260928 02:45:25.367933 21934 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39251
I20260928 02:45:25.368160 21934 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:25.368671 21934 heartbeater.cc:511] Master 127.2.123.62:39251 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.123.2@KRBTEST.COM; defaulting to no policy
I20260928 02:45:25.370159 21743 ts_manager.cc:194] Registered new tserver with Master: d1b76c42e05647928e5138aa4f9f3050 (127.2.123.1:40877)
I20260928 02:45:25.371379 21743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.1@KRBTEST.COM'} at 127.2.123.1:39531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
Entry for principal kudu/127.2.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab.
I20260928 02:45:25.386898 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39251
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:25.493347 21939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:25.493614 21939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:25.493656 21939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:25.497575 21939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:25.497632 21939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:25.497689 21939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:45:25.502990 21939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.2.keytab
--principal=kudu/127.2.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:25.503957 21939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:25.504921 21939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:25.515046 21945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:25.515077 21944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:25.515240 21939 server_base.cc:1034] running on GCE node
W20260928 02:45:25.515348 21947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:25.515771 21939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:25.516438 21939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:25.517616 21939 hybrid_clock.cc:648] HybridClock initialized: now 1790563525517555 us; error 77 us; skew 500 ppm
Sep 28 02:45:25 dist-test-slave-rqvb krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:25.520380 21939 init.cc:377] Logged in from keytab as kudu/127.2.123.2@KRBTEST.COM (short username kudu)
I20260928 02:45:25.521260 21939 webserver.cc:460] Webserver started at http://127.2.123.2:40315/ using document root <none> and password file <none>
I20260928 02:45:25.521781 21939 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:25.521847 21939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:25.522099 21939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:25.524137 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "95b7460c418e491d9d8fc4e2deaa008c"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "5a92b05ef902c3be139bfbc5aae5629d"
server_key_iv: "ff8de27a9cfc88737de6c475ee267df5"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.524689 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "95b7460c418e491d9d8fc4e2deaa008c"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "5a92b05ef902c3be139bfbc5aae5629d"
server_key_iv: "ff8de27a9cfc88737de6c475ee267df5"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.528525 21939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 02:45:25.530983 21954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.532301 21939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:45:25.532409 21939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "95b7460c418e491d9d8fc4e2deaa008c"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "5a92b05ef902c3be139bfbc5aae5629d"
server_key_iv: "ff8de27a9cfc88737de6c475ee267df5"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.532500 21939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:25.563882 21939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:25.567039 21939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:25.567229 21939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:25.567816 21939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:25.568648 21939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:25.568696 21939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.568750 21939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:25.568779 21939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.577049 21939 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:44117
I20260928 02:45:25.577090 22067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:44117 every 8 connection(s)
I20260928 02:45:25.577909 21939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:45:25.582964 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 21939
I20260928 02:45:25.583076 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:45:25.583362 2540 external_mini_cluster.cc:1428] Setting key 70b89a74d328e99439b1d1ef80cf48b7
Sep 28 02:45:25 dist-test-slave-rqvb krb5kdc[21698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.2@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.3@KRBTEST.COM; defaulting to no policy
I20260928 02:45:25.592064 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.579737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:46857 (local address 127.2.123.62:39251)
0928 02:45:25.579880 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.579885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.580725 (+ 840us) server_negotiation.cc:407] Connection header received
0928 02:45:25.582023 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.582026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.582079 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.582156 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:25.583365 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.583943 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.585466 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.585765 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.585863 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.588484 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:25.588487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:25.588489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:25.588515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:25.589928 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.590489 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.590493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.590495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.590585 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.590917 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.590920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.590922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.591127 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:25.591230 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:25.591742 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 02:45:25.591872 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:45:25.592908 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.580144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39251 (local address 127.2.123.2:46857)
0928 02:45:25.580572 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.580616 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.581782 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:25.582339 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:25.582347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:25.582700 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:25.583218 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.583241 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.584137 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.584142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.585333 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.585344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.585914 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.585918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.586182 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.586795 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:25.586814 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:25.588301 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:25.590093 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:25.590100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:25.590117 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:25.590387 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:25.590676 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:25.590679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:25.590682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:25.590827 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:25.591231 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:25.591238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:25.591491 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 02:45:25.591681 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 02:45:25.592020 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":116,"threads_started":1}
I20260928 02:45:25.594066 22068 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.3@KRBTEST.COM" created.
I20260928 02:45:25.594341 22068 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:25.594808 22068 heartbeater.cc:511] Master 127.2.123.62:39251 requested a full tablet report, sending...
I20260928 02:45:25.595892 21743 ts_manager.cc:194] Registered new tserver with Master: 95b7460c418e491d9d8fc4e2deaa008c (127.2.123.2:44117)
I20260928 02:45:25.596525 21743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.2@KRBTEST.COM'} at 127.2.123.2:46857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
Entry for principal kudu/127.2.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab.
I20260928 02:45:25.609701 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39251
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:25.716367 22073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:25.716624 22073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:25.716665 22073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:25.720674 22073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:25.720733 22073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:25.720816 22073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:45:25.726090 22073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.3.keytab
--principal=kudu/127.2.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:39251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:25.727097 22073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:25.728134 22073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:25.737594 22081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:25.737583 22078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:25.737563 22079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:25.737645 22073 server_base.cc:1034] running on GCE node
I20260928 02:45:25.738255 22073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:25.738966 22073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:25.740125 22073 hybrid_clock.cc:648] HybridClock initialized: now 1790563525740107 us; error 32 us; skew 500 ppm
Sep 28 02:45:25 dist-test-slave-rqvb krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:25.742940 22073 init.cc:377] Logged in from keytab as kudu/127.2.123.3@KRBTEST.COM (short username kudu)
I20260928 02:45:25.743836 22073 webserver.cc:460] Webserver started at http://127.2.123.3:38729/ using document root <none> and password file <none>
I20260928 02:45:25.744367 22073 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:25.744427 22073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:25.744611 22073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:25.746650 22073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "30917bdd65a8463bb3e53b306460513e"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "3ce193bec2c24d22dfd3a501eb920b3d"
server_key_iv: "3a01d292e9cd8e3068c850eaeee0b795"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.747180 22073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "30917bdd65a8463bb3e53b306460513e"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "3ce193bec2c24d22dfd3a501eb920b3d"
server_key_iv: "3a01d292e9cd8e3068c850eaeee0b795"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.750913 22073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:45:25.753460 22088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.754995 22073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:45:25.755100 22073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "30917bdd65a8463bb3e53b306460513e"
format_stamp: "Formatted at 2026-09-28 02:45:25 on dist-test-slave-rqvb"
server_key: "3ce193bec2c24d22dfd3a501eb920b3d"
server_key_iv: "3a01d292e9cd8e3068c850eaeee0b795"
server_key_version: "encryptionkey@0"
I20260928 02:45:25.755205 22073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:25.792858 22073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:25.795985 22073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:25.796166 22073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:25.796756 22073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:25.797570 22073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:25.797617 22073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.797669 22073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:25.797699 22073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:25.805779 22073 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:38891
I20260928 02:45:25.805863 22201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:38891 every 8 connection(s)
I20260928 02:45:25.806699 22073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:45:25.816109 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 22073
I20260928 02:45:25.816334 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
Sep 28 02:45:25 dist-test-slave-rqvb krb5kdc[21698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563525, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.3@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:25.816636 2540 external_mini_cluster.cc:1428] Setting key 16cbb994e8e86708f5f98f2bc1b82117
I20260928 02:45:25.821017 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.808482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:60389 (local address 127.2.123.62:39251)
0928 02:45:25.808616 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.808620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.809497 (+ 877us) server_negotiation.cc:407] Connection header received
0928 02:45:25.811161 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.811166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.811222 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.811313 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:25.812731 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.813321 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.814497 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.814724 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.814807 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.817352 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:25.817355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:25.817357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:25.817379 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:25.818764 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.819493 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.819496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.819498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.819581 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.819920 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.819923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.819925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.820112 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:25.820197 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:25.820710 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 02:45:25.820843 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:45:25.821827 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.808833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:39251 (local address 127.2.123.3:60389)
0928 02:45:25.809339 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.809399 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.810911 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:25.811500 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:25.811510 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:25.812000 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:25.812575 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.812589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.813426 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.813429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.814383 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.814392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.814842 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.814844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.814977 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.815469 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:25.815484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:25.817018 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:25.818869 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:25.818875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:25.818890 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:25.819374 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:25.819675 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:25.819678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:25.819680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:25.819831 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:25.820210 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:25.820217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:25.820457 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:45:25.820622 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:45:25.820997 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260928 02:45:25.822948 22202 heartbeater.cc:347] Connected to a master server at 127.2.123.62:39251
I20260928 02:45:25.823170 22202 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:25.823634 22202 heartbeater.cc:511] Master 127.2.123.62:39251 requested a full tablet report, sending...
I20260928 02:45:25.824601 21743 ts_manager.cc:194] Registered new tserver with Master: 30917bdd65a8463bb3e53b306460513e (127.2.123.3:38891)
I20260928 02:45:25.825162 21743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.123.3@KRBTEST.COM'} at 127.2.123.3:60389
I20260928 02:45:25.830592 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:45:25.840018 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.832638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.2.123.62:39251)
0928 02:45:25.832786 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.832790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.832887 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:45:25.833017 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.833020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.833065 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.833137 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:25.833857 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.834439 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.835297 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.835517 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.835612 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.836911 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:25.836915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:25.836917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:25.836944 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:25.838397 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.838799 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.838803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.838805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.838885 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:25.839132 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:25.839135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:25.839137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:25.839382 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:25.839499 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:25.839739 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:45:25.839861 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:45:25.843139 21743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:45:25.846216 21743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:45:25.861371 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.855052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:38891 (local address 127.0.0.1:38596)
0928 02:45:25.855835 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.855890 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.856031 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:25.857335 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:25.857338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:25.857356 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:25.857430 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.857435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.858822 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.858825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.859858 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.859867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.860741 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.860744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.860877 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.860917 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:45:25.861021 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:45:25.861145 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"spinlock_wait_cycles":69248,"thread_start_us":85,"threads_started":1}
I20260928 02:45:25.861408 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.856235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.2.123.1:40877)
0928 02:45:25.856507 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.856521 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.856543 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:45:25.856620 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.856625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.856827 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.856975 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:25.857387 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.858295 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.860019 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.860867 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.860974 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.861117 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:25.861169 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:45:25.861258 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
I20260928 02:45:25.861485 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.855805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.2.123.3:38891)
0928 02:45:25.856767 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.856788 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.856809 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:45:25.856876 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.856880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.857041 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.857182 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:25.857550 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.858709 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.859988 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.860618 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.860717 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.860890 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:25.861260 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:45:25.861351 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":159,"threads_started":1}
I20260928 02:45:25.861406 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.855052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:44117 (local address 127.0.0.1:42504)
0928 02:45:25.855835 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.855890 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.856029 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:25.856725 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:25.856732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:25.856757 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:25.856905 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.856912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.858575 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.858578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.859868 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.859879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.861004 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.861007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.861092 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.861114 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:45:25.861162 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:45:25.861233 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":43,"spinlock_wait_cycles":16000,"thread_start_us":113,"threads_started":1}
I20260928 02:45:25.861826 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.855202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:40877 (local address 127.0.0.1:42488)
0928 02:45:25.855835 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.855903 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.856029 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:25.857133 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:25.857136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:25.857153 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:25.857249 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.857255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.858415 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.858419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.859892 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.859899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.860966 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.860970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.861056 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.861077 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:45:25.861127 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:45:25.861188 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":45,"spinlock_wait_cycles":3072,"thread_start_us":90,"threads_started":1}
I20260928 02:45:25.862440 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.855305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.2.123.2:44117)
0928 02:45:25.856301 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.856310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.856330 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:45:25.856395 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.856399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.856563 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.857058 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:25.857247 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.858450 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.860019 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.860867 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.862018 (+ 1151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:25.862171 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:25.862220 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:45:25.862313 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":109,"threads_started":1}
I20260928 02:45:25.864630 21868 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f4b0a79a24a8da85636af2e4e1c92 (DEFAULT_TABLE table=test-table [id=d3cc41b62f1540699f873224f84d9f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:25.864670 22136 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f4b0a79a24a8da85636af2e4e1c92 (DEFAULT_TABLE table=test-table [id=d3cc41b62f1540699f873224f84d9f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:25.864995 22002 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f4b0a79a24a8da85636af2e4e1c92 (DEFAULT_TABLE table=test-table [id=d3cc41b62f1540699f873224f84d9f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:25.865628 21868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f4b0a79a24a8da85636af2e4e1c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:25.865733 22136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f4b0a79a24a8da85636af2e4e1c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:25.865811 22002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f4b0a79a24a8da85636af2e4e1c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:25.873852 22221 tablet_bootstrap.cc:492] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Bootstrap starting.
I20260928 02:45:25.874320 22220 tablet_bootstrap.cc:492] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: Bootstrap starting.
I20260928 02:45:25.876036 22219 tablet_bootstrap.cc:492] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050: Bootstrap starting.
I20260928 02:45:25.876228 22221 tablet_bootstrap.cc:654] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:25.876634 22220 tablet_bootstrap.cc:654] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:25.877465 22220 log.cc:824] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:25.877465 22221 log.cc:824] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:25.878373 22219 tablet_bootstrap.cc:654] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:25.879292 22219 log.cc:824] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:25.880612 22221 tablet_bootstrap.cc:492] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: No bootstrap required, opened a new log
I20260928 02:45:25.880635 22220 tablet_bootstrap.cc:492] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: No bootstrap required, opened a new log
I20260928 02:45:25.880825 22221 ts_tablet_manager.cc:1397] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:45:25.880842 22220 ts_tablet_manager.cc:1397] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:45:25.881318 22219 tablet_bootstrap.cc:492] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050: No bootstrap required, opened a new log
I20260928 02:45:25.881494 22219 ts_tablet_manager.cc:1397] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:45:25.884788 22220 raft_consensus.cc:348] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:25.884783 22221 raft_consensus.cc:348] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:25.884900 22219 raft_consensus.cc:348] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:25.885087 22221 raft_consensus.cc:374] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:25.885087 22220 raft_consensus.cc:374] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:25.885092 22219 raft_consensus.cc:374] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:25.885144 22221 raft_consensus.cc:729] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30917bdd65a8463bb3e53b306460513e, State: Initialized, Role: FOLLOWER
I20260928 02:45:25.885144 22220 raft_consensus.cc:729] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b7460c418e491d9d8fc4e2deaa008c, State: Initialized, Role: FOLLOWER
I20260928 02:45:25.885144 22219 raft_consensus.cc:729] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b76c42e05647928e5138aa4f9f3050, State: Initialized, Role: FOLLOWER
I20260928 02:45:25.885591 22219 consensus_queue.cc:260] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:25.885601 22220 consensus_queue.cc:260] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:25.885601 22221 consensus_queue.cc:260] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:25.886276 21934 heartbeater.cc:503] Master 127.2.123.62:39251 was elected leader, sending a full tablet report...
I20260928 02:45:25.886554 22219 ts_tablet_manager.cc:1428] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:45:25.886587 22221 ts_tablet_manager.cc:1428] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260928 02:45:25.886936 22202 heartbeater.cc:503] Master 127.2.123.62:39251 was elected leader, sending a full tablet report...
I20260928 02:45:25.887238 22068 heartbeater.cc:503] Master 127.2.123.62:39251 was elected leader, sending a full tablet report...
I20260928 02:45:25.887687 22220 ts_tablet_manager.cc:1428] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:45:25.995220 22226 raft_consensus.cc:482] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:25.995432 22226 raft_consensus.cc:504] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:25.996620 22226 leader_election.cc:288] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30917bdd65a8463bb3e53b306460513e (127.2.123.3:38891), d1b76c42e05647928e5138aa4f9f3050 (127.2.123.1:40877)
I20260928 02:45:25.997321 22227 raft_consensus.cc:482] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:25.997529 22227 raft_consensus.cc:504] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:25.999315 22227 leader_election.cc:288] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95b7460c418e491d9d8fc4e2deaa008c (127.2.123.2:44117), d1b76c42e05647928e5138aa4f9f3050 (127.2.123.1:40877)
I20260928 02:45:26.001005 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.996852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:38891 (local address 127.2.123.2:53453)
0928 02:45:25.997039 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.997054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.997166 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:25.997457 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:25.997460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:25.997477 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:25.997597 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.997603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.998512 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.998515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:25.999694 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.999703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.000567 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.000571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.000686 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.000710 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.000779 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.000866 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 02:45:26.001478 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.996954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:53453 (local address 127.2.123.3:38891)
0928 02:45:25.997064 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.997069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.997249 (+ 180us) server_negotiation.cc:407] Connection header received
0928 02:45:25.997315 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.997318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.997356 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.997415 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:25.997714 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.998398 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.999815 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.000358 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.000448 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.000511 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.001158 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.001369 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":49}
I20260928 02:45:26.001569 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.997008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:60283 (local address 127.2.123.1:40877)
0928 02:45:25.997134 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.997138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.997432 (+ 294us) server_negotiation.cc:407] Connection header received
0928 02:45:25.997849 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.997852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.997905 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.998017 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:25.998297 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.998865 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.000269 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.000924 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.001008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.001066 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.001370 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.001441 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":51}
I20260928 02:45:26.001922 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.996922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:40877 (local address 127.2.123.2:60283)
0928 02:45:25.997335 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.997348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.997600 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:25.998032 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:25.998034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:25.998052 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:25.998184 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:25.998192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:25.998992 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:25.998996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.000137 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.000145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.001142 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.001145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.001236 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.001259 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.001697 (+ 438us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.001770 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":148,"threads_started":1}
I20260928 02:45:26.002107 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f4b0a79a24a8da85636af2e4e1c92" candidate_uuid: "95b7460c418e491d9d8fc4e2deaa008c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30917bdd65a8463bb3e53b306460513e" is_pre_election: true
I20260928 02:45:26.002336 22156 raft_consensus.cc:2454] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b7460c418e491d9d8fc4e2deaa008c in term 0.
I20260928 02:45:26.002583 21888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f4b0a79a24a8da85636af2e4e1c92" candidate_uuid: "95b7460c418e491d9d8fc4e2deaa008c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b76c42e05647928e5138aa4f9f3050" is_pre_election: true
I20260928 02:45:26.002801 21956 leader_election.cc:302] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30917bdd65a8463bb3e53b306460513e, 95b7460c418e491d9d8fc4e2deaa008c; no voters:
I20260928 02:45:26.002873 21888 raft_consensus.cc:2454] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b7460c418e491d9d8fc4e2deaa008c in term 0.
I20260928 02:45:26.003077 22226 raft_consensus.cc:2781] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:45:26.003140 22226 raft_consensus.cc:482] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:45:26.003175 22226 raft_consensus.cc:3037] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:26.003507 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.998992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:44117 (local address 127.2.123.3:58377)
0928 02:45:25.999226 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.999238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.999321 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:26.000073 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:26.000077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:26.000093 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:26.000203 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.000209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.001115 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.001118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.002269 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.002279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.003130 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.003134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.003225 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.003248 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.003322 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.003388 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1}
I20260928 02:45:26.003919 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.999751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:58377 (local address 127.2.123.2:44117)
0928 02:45:25.999845 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.999849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.999861 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:45:25.999896 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.999898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:25.999931 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:25.999989 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:26.000313 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.000998 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.002410 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.002927 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.003022 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.003074 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.003738 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.003811 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:45:26.004374 22022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f4b0a79a24a8da85636af2e4e1c92" candidate_uuid: "30917bdd65a8463bb3e53b306460513e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b7460c418e491d9d8fc4e2deaa008c" is_pre_election: true
I20260928 02:45:26.004459 22226 raft_consensus.cc:504] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:26.004642 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.999496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:49617 (local address 127.2.123.1:40877)
0928 02:45:25.999922 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:25.999929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:25.999940 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:45:25.999986 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:25.999989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.000027 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.000204 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:26.000568 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.001497 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.003517 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.004091 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.004189 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.004244 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.004447 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.004517 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":359,"thread_start_us":113,"threads_started":1}
I20260928 02:45:26.004734 22022 raft_consensus.cc:2379] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30917bdd65a8463bb3e53b306460513e in current term 1: Already voted for candidate 95b7460c418e491d9d8fc4e2deaa008c in this term.
I20260928 02:45:26.004856 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:25.999015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:40877 (local address 127.2.123.3:49617)
0928 02:45:25.999560 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:25.999572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:25.999653 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:26.000153 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:26.000157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:26.000174 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:26.000452 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.000457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.002221 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.002225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.003262 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.003272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.004215 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.004219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.004312 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.004336 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.004412 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.004541 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":144,"thread_start_us":116,"threads_started":1}
I20260928 02:45:26.005163 22156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f4b0a79a24a8da85636af2e4e1c92" candidate_uuid: "95b7460c418e491d9d8fc4e2deaa008c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30917bdd65a8463bb3e53b306460513e"
I20260928 02:45:26.005260 21888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f4b0a79a24a8da85636af2e4e1c92" candidate_uuid: "95b7460c418e491d9d8fc4e2deaa008c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b76c42e05647928e5138aa4f9f3050"
I20260928 02:45:26.005276 22156 raft_consensus.cc:3037] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:26.005389 21888 raft_consensus.cc:3037] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:26.006299 22156 raft_consensus.cc:2454] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b7460c418e491d9d8fc4e2deaa008c in term 1.
I20260928 02:45:26.006471 21888 raft_consensus.cc:2454] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b7460c418e491d9d8fc4e2deaa008c in term 1.
I20260928 02:45:26.006536 22226 leader_election.cc:288] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [CANDIDATE]: Term 1 election: Requested vote from peers 30917bdd65a8463bb3e53b306460513e (127.2.123.3:38891), d1b76c42e05647928e5138aa4f9f3050 (127.2.123.1:40877)
I20260928 02:45:26.006572 21956 leader_election.cc:302] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30917bdd65a8463bb3e53b306460513e, 95b7460c418e491d9d8fc4e2deaa008c; no voters:
I20260928 02:45:26.006848 22226 raft_consensus.cc:2781] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:26.006948 21888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f4b0a79a24a8da85636af2e4e1c92" candidate_uuid: "30917bdd65a8463bb3e53b306460513e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b76c42e05647928e5138aa4f9f3050" is_pre_election: true
I20260928 02:45:26.007058 22226 raft_consensus.cc:686] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 1 LEADER]: Becoming Leader. State: Replica: 95b7460c418e491d9d8fc4e2deaa008c, State: Running, Role: LEADER
I20260928 02:45:26.007110 21888 raft_consensus.cc:2379] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30917bdd65a8463bb3e53b306460513e in current term 1: Already voted for candidate 95b7460c418e491d9d8fc4e2deaa008c in this term.
I20260928 02:45:26.007388 22226 consensus_queue.cc:237] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } }
I20260928 02:45:26.007548 22091 leader_election.cc:302] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30917bdd65a8463bb3e53b306460513e; no voters: 95b7460c418e491d9d8fc4e2deaa008c, d1b76c42e05647928e5138aa4f9f3050
I20260928 02:45:26.007819 22227 raft_consensus.cc:2726] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:45:26.010716 21743 catalog_manager.cc:5495] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c reported cstate change: term changed from 0 to 1, leader changed from <none> to 95b7460c418e491d9d8fc4e2deaa008c (127.2.123.2). New cstate: current_term: 1 leader_uuid: "95b7460c418e491d9d8fc4e2deaa008c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b7460c418e491d9d8fc4e2deaa008c" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 } health_report { overall_health: UNKNOWN } } }
I20260928 02:45:26.056414 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.052834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.2.123.2:44117)
0928 02:45:26.052950 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.052954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.052966 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:45:26.053044 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.053046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.053100 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.053189 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:26.053460 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.054053 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.054911 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.055129 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.055213 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.055498 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:26.055924 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:26.056048 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.056235 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.056290 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 02:45:26.057734 22203 tablet.cc:2369] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:26.062328 21888 raft_consensus.cc:1261] T ba4f4b0a79a24a8da85636af2e4e1c92 P d1b76c42e05647928e5138aa4f9f3050 [term 1 FOLLOWER]: Refusing update from remote peer 95b7460c418e491d9d8fc4e2deaa008c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:26.062340 22156 raft_consensus.cc:1261] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 1 FOLLOWER]: Refusing update from remote peer 95b7460c418e491d9d8fc4e2deaa008c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:26.062938 22231 consensus_queue.cc:1035] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1b76c42e05647928e5138aa4f9f3050" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:26.063105 22226 consensus_queue.cc:1035] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30917bdd65a8463bb3e53b306460513e" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:26.071072 22236 mvcc.cc:205] Tried to move back new op lower bound from 7334148202744279040 to 7334148202528964608. Current Snapshot: MvccSnapshot[applied={T|T < 7334148202744279040}]
I20260928 02:45:26.071470 22235 mvcc.cc:205] Tried to move back new op lower bound from 7334148202744279040 to 7334148202528964608. Current Snapshot: MvccSnapshot[applied={T|T < 7334148202744279040}]
I20260928 02:45:26.078154 21742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:26.078359 21742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:26.080662 21742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 122ee37aa8eb48c8baa4c9dd9bd4da36: Sending DeleteTablet for 3 replicas of tablet ba4f4b0a79a24a8da85636af2e4e1c92
I20260928 02:45:26.081398 22002 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4f4b0a79a24a8da85636af2e4e1c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:26 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:42504
I20260928 02:45:26.081458 22136 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4f4b0a79a24a8da85636af2e4e1c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:26 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:38596
I20260928 02:45:26.081532 21868 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4f4b0a79a24a8da85636af2e4e1c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:26 UTC) from {username='kudu', principal='kudu/127.2.123.62@KRBTEST.COM'} at 127.0.0.1:42488
I20260928 02:45:26.081911 22244 tablet_replica.cc:330] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: stopping tablet replica
I20260928 02:45:26.082223 22244 raft_consensus.cc:2229] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:45:26.082572 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 21805
I20260928 02:45:26.082563 22244 raft_consensus.cc:2258] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:26.086647 22244 ts_tablet_manager.cc:1905] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:26.089517 22244 ts_tablet_manager.cc:1918] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:26.089624 22244 log.cc:1196] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/ba4f4b0a79a24a8da85636af2e4e1c92
I20260928 02:45:26.089915 22244 ts_tablet_manager.cc:1939] T ba4f4b0a79a24a8da85636af2e4e1c92 P 95b7460c418e491d9d8fc4e2deaa008c: Deleting consensus metadata
I20260928 02:45:26.091171 21727 catalog_manager.cc:4841] TS 95b7460c418e491d9d8fc4e2deaa008c (127.2.123.2:44117): tablet ba4f4b0a79a24a8da85636af2e4e1c92 (table test-table [id=d3cc41b62f1540699f873224f84d9f42]) successfully deleted
I20260928 02:45:26.094830 22245 tablet_replica.cc:330] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: stopping tablet replica
I20260928 02:45:26.095160 22245 raft_consensus.cc:2229] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:26.095489 22245 raft_consensus.cc:2258] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:45:26.095969 21729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:40877 (error 108)
I20260928 02:45:26.096757 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 21939
I20260928 02:45:26.097029 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.096718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:40877 (local address 127.0.0.1:42500)
0928 02:45:26.096890 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:26.096953 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:40877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 02:45:26.097285 21729 catalog_manager.cc:4584] TS d1b76c42e05647928e5138aa4f9f3050 (127.2.123.1:40877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba4f4b0a79a24a8da85636af2e4e1c92: Network error: Client connection negotiation failed: client connection to 127.2.123.1:40877: connect: Connection refused (error 111)
I20260928 02:45:26.097553 22245 ts_tablet_manager.cc:1905] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:26.099864 22245 ts_tablet_manager.cc:1918] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:26.099946 22245 log.cc:1196] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/ba4f4b0a79a24a8da85636af2e4e1c92
I20260928 02:45:26.100188 22245 ts_tablet_manager.cc:1939] T ba4f4b0a79a24a8da85636af2e4e1c92 P 30917bdd65a8463bb3e53b306460513e: Deleting consensus metadata
I20260928 02:45:26.101272 21728 catalog_manager.cc:4841] TS 30917bdd65a8463bb3e53b306460513e (127.2.123.3:38891): tablet ba4f4b0a79a24a8da85636af2e4e1c92 (table test-table [id=d3cc41b62f1540699f873224f84d9f42]) successfully deleted
I20260928 02:45:26.102051 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 22073
I20260928 02:45:26.106892 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 21711
2026-09-28T02:45:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2272 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:26Z Disabled control of system clock
I20260928 02:45:26.135159 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33187
--webserver_interface=127.2.123.62
--webserver_port=0
--builtin_ntp_servers=127.2.123.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:45:26.241393 22251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:26.241652 22251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:26.241684 22251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:26.245584 22251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:26.245641 22251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:26.245656 22251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:26.245671 22251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:26.245683 22251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:26.250312 22251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:33187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:26.251278 22251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:26.252557 22251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:26.260641 22257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:26.260701 22256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:26.260861 22251 server_base.cc:1034] running on GCE node
W20260928 02:45:26.260632 22259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:26.261547 22251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:26.262641 22251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:26.263830 22251 hybrid_clock.cc:648] HybridClock initialized: now 1790563526263811 us; error 32 us; skew 500 ppm
I20260928 02:45:26.265718 22251 webserver.cc:460] Webserver started at http://127.2.123.62:34935/ using document root <none> and password file <none>
I20260928 02:45:26.266316 22251 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:26.266378 22251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:26.266553 22251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:26.268860 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "a6e54a0403a24ae89ec1246f243c4cf1"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "d9f1de4c9a15fd03f0f7ad88b9a8de2a"
server_key_iv: "65005747216e98fab153ebf3cd0047db"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.269395 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "a6e54a0403a24ae89ec1246f243c4cf1"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "d9f1de4c9a15fd03f0f7ad88b9a8de2a"
server_key_iv: "65005747216e98fab153ebf3cd0047db"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.273273 22251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:45:26.275867 22265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.277143 22251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:45:26.277307 22251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "a6e54a0403a24ae89ec1246f243c4cf1"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "d9f1de4c9a15fd03f0f7ad88b9a8de2a"
server_key_iv: "65005747216e98fab153ebf3cd0047db"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.277421 22251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:26.296711 22251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:26.297428 22251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:26.297605 22251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:26.304039 22251 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:33187
I20260928 02:45:26.304054 22317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:33187 every 8 connection(s)
I20260928 02:45:26.304957 22251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:26.308207 22318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:26.310725 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 22251
I20260928 02:45:26.310834 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:26.311089 2540 external_mini_cluster.cc:1428] Setting key f3dbf466b03fd729dadd87a29382f400
I20260928 02:45:26.314183 22318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1: Bootstrap starting.
I20260928 02:45:26.316339 22318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:26.317056 22318 log.cc:824] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:26.318940 22318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1: No bootstrap required, opened a new log
I20260928 02:45:26.320369 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.312537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.2.123.62:33187)
0928 02:45:26.313225 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.313244 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.313281 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:45:26.313902 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.313920 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.314382 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.314746 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:26.315154 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.316100 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.316978 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.317461 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.317604 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.318443 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:26.318448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:26.318462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:26.318842 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:26.318895 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:26.318914 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:26.318997 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.319167 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.319457 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260928 02:45:26.322722 22318 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e54a0403a24ae89ec1246f243c4cf1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33187 } }
I20260928 02:45:26.322937 22318 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:26.322981 22318 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e54a0403a24ae89ec1246f243c4cf1, State: Initialized, Role: FOLLOWER
I20260928 02:45:26.323329 22318 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e54a0403a24ae89ec1246f243c4cf1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33187 } }
I20260928 02:45:26.323436 22318 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:26.323477 22318 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:26.323541 22318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:26.324362 22318 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e54a0403a24ae89ec1246f243c4cf1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33187 } }
I20260928 02:45:26.324677 22318 leader_election.cc:302] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6e54a0403a24ae89ec1246f243c4cf1; no voters:
I20260928 02:45:26.324941 22318 leader_election.cc:288] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:26.325023 22323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:26.325186 22323 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [term 1 LEADER]: Becoming Leader. State: Replica: a6e54a0403a24ae89ec1246f243c4cf1, State: Running, Role: LEADER
I20260928 02:45:26.325495 22323 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e54a0403a24ae89ec1246f243c4cf1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33187 } }
I20260928 02:45:26.326208 22318 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:26.326767 22324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6e54a0403a24ae89ec1246f243c4cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e54a0403a24ae89ec1246f243c4cf1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33187 } } }
I20260928 02:45:26.326812 22325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6e54a0403a24ae89ec1246f243c4cf1. Latest consensus state: current_term: 1 leader_uuid: "a6e54a0403a24ae89ec1246f243c4cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e54a0403a24ae89ec1246f243c4cf1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33187 } } }
I20260928 02:45:26.327008 22325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:26.327035 22324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:26.327451 22332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:26.330520 22332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:26.335578 22332 catalog_manager.cc:1347] Generated new cluster ID: 6c7744676d484e23addf276db578375a
I20260928 02:45:26.335645 22332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:26.345995 22332 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:26.346912 22332 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:26.353608 22332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1: Generated new TSK 0
I20260928 02:45:26.354313 22332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:26.362628 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.1:0
--local_ip_for_outbound_sockets=127.2.123.1
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33187
--builtin_ntp_servers=127.2.123.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:45:26.468427 22342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:26.468670 22342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:26.468719 22342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:26.472687 22342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:26.472743 22342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:26.472801 22342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.1
I20260928 02:45:26.477516 22342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.123.1
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.1
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:26.478595 22342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:26.479913 22342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:26.489733 22350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:26.489778 22342 server_base.cc:1034] running on GCE node
W20260928 02:45:26.489693 22347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:26.489706 22348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:26.490540 22342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:26.491197 22342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:26.492367 22342 hybrid_clock.cc:648] HybridClock initialized: now 1790563526492342 us; error 34 us; skew 500 ppm
I20260928 02:45:26.494114 22342 webserver.cc:460] Webserver started at http://127.2.123.1:43761/ using document root <none> and password file <none>
I20260928 02:45:26.494664 22342 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:26.494731 22342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:26.494915 22342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:26.496942 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/data/instance:
uuid: "f20be496ba24466b803637a2ac753d8e"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "e03dd7747e9ac9291e2e8a5a799a9ed3"
server_key_iv: "58770343f956ef50300f41779ed91ea3"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.497464 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance:
uuid: "f20be496ba24466b803637a2ac753d8e"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "e03dd7747e9ac9291e2e8a5a799a9ed3"
server_key_iv: "58770343f956ef50300f41779ed91ea3"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.500985 22342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:45:26.503386 22356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.504732 22342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:26.504843 22342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/wal
uuid: "f20be496ba24466b803637a2ac753d8e"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "e03dd7747e9ac9291e2e8a5a799a9ed3"
server_key_iv: "58770343f956ef50300f41779ed91ea3"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.504940 22342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:26.519400 22342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:26.520092 22342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:26.520272 22342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:26.520872 22342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:26.521821 22342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:26.521873 22342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.521926 22342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:26.521953 22342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.530658 22342 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.1:43995
I20260928 02:45:26.530683 22469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.1:43995 every 8 connection(s)
I20260928 02:45:26.531539 22342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/data/info.pb
I20260928 02:45:26.538683 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 22342
I20260928 02:45:26.538828 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-0/wal/instance
I20260928 02:45:26.539117 2540 external_mini_cluster.cc:1428] Setting key ca17fd5e54b0e3033404a07053b0b4f9
I20260928 02:45:26.541203 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.2:0
--local_ip_for_outbound_sockets=127.2.123.2
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33187
--builtin_ntp_servers=127.2.123.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:45:26.541419 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.533403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.1:34239 (local address 127.2.123.62:33187)
0928 02:45:26.533543 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.533547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.534681 (+ 1134us) server_negotiation.cc:407] Connection header received
0928 02:45:26.536084 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.536087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.536199 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.536281 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:26.537225 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.537778 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.539281 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.539538 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.539658 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.540879 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:26.540881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:26.540884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:26.540895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:26.540916 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:26.540923 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:26.540984 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.541147 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.541277 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:45:26.542327 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.533844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33187 (local address 127.2.123.1:34239)
0928 02:45:26.534535 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:26.534575 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:26.535871 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:26.536489 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:26.536495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:26.536580 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:26.537054 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.537066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.537904 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.537906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.539134 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.539146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.539710 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.539714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.539908 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.540532 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:26.540534 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:26.540603 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:26.540680 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:26.540997 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:26.541007 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:26.541013 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.541133 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.541439 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":156,"threads_started":1}
I20260928 02:45:26.543530 22470 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33187
I20260928 02:45:26.543749 22470 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:26.544245 22470 heartbeater.cc:511] Master 127.2.123.62:33187 requested a full tablet report, sending...
I20260928 02:45:26.545845 22282 ts_manager.cc:194] Registered new tserver with Master: f20be496ba24466b803637a2ac753d8e (127.2.123.1:43995)
I20260928 02:45:26.547464 22282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.1:34239
W20260928 02:45:26.648777 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:26.649029 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:26.649066 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:26.653033 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:26.653090 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:26.653142 22473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.2
I20260928 02:45:26.657778 22473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.123.2
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.2
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:26.658803 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:26.660064 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:26.669155 22478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:26.669191 22473 server_base.cc:1034] running on GCE node
W20260928 02:45:26.669188 22479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:26.669139 22481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:26.669801 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:26.670475 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:26.671677 22473 hybrid_clock.cc:648] HybridClock initialized: now 1790563526671653 us; error 33 us; skew 500 ppm
I20260928 02:45:26.673595 22473 webserver.cc:460] Webserver started at http://127.2.123.2:33839/ using document root <none> and password file <none>
I20260928 02:45:26.674254 22473 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:26.674337 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:26.674618 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:26.676705 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/data/instance:
uuid: "b3bb522c49c442af8346bb494cc92366"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "03349fb61b669203cbed8b8c075283ca"
server_key_iv: "33e4a4f035e7e90fd4ed98ddbcd4af99"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.677271 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance:
uuid: "b3bb522c49c442af8346bb494cc92366"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "03349fb61b669203cbed8b8c075283ca"
server_key_iv: "33e4a4f035e7e90fd4ed98ddbcd4af99"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.680961 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:45:26.683717 22487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.684942 22473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:26.685052 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/wal
uuid: "b3bb522c49c442af8346bb494cc92366"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "03349fb61b669203cbed8b8c075283ca"
server_key_iv: "33e4a4f035e7e90fd4ed98ddbcd4af99"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.685258 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:26.702210 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:26.702939 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:26.703120 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:26.703711 22473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:26.704634 22473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:26.704687 22473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.704730 22473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:26.704757 22473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.714051 22473 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.2:43689
I20260928 02:45:26.714074 22600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.2:43689 every 8 connection(s)
I20260928 02:45:26.714993 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/data/info.pb
I20260928 02:45:26.717067 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 22473
I20260928 02:45:26.717172 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/wal/instance
I20260928 02:45:26.717460 2540 external_mini_cluster.cc:1428] Setting key 291eb59c314cb829e1c7a1a62d78a9e0
I20260928 02:45:26.719401 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.123.3:0
--local_ip_for_outbound_sockets=127.2.123.3
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33187
--builtin_ntp_servers=127.2.123.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:45:26.724895 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.716797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:48933 (local address 127.2.123.62:33187)
0928 02:45:26.716948 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.716953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.718139 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 02:45:26.719482 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.719485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.719537 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.719625 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:26.720928 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.721742 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.722958 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.723293 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.723407 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.724411 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:26.724415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:26.724418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:26.724432 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:26.724454 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:26.724465 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:26.724542 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.724684 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.724747 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:45:26.725737 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.717244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33187 (local address 127.2.123.2:48933)
0928 02:45:26.717988 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:26.718042 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:26.719272 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:26.719927 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:26.719945 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:26.720058 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:26.720722 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.720744 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.721877 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.721880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.722828 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.722839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.723443 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.723446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.723592 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.724144 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:26.724146 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:26.724195 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:26.724265 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:26.724549 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:26.724556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:26.724561 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.724656 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.724884 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260928 02:45:26.726872 22601 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33187
I20260928 02:45:26.727144 22601 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:26.727799 22601 heartbeater.cc:511] Master 127.2.123.62:33187 requested a full tablet report, sending...
I20260928 02:45:26.728988 22282 ts_manager.cc:194] Registered new tserver with Master: b3bb522c49c442af8346bb494cc92366 (127.2.123.2:43689)
I20260928 02:45:26.729553 22282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.2:48933
W20260928 02:45:26.843034 22604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:26.843312 22604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:26.843348 22604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:26.847435 22604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:26.847498 22604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:26.847558 22604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.123.3
I20260928 02:45:26.852378 22604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.123.3
--webserver_port=0
--tserver_master_addrs=127.2.123.62:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.123.3
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:26.853462 22604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:26.854913 22604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:26.864146 22609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:26.864176 22612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:26.864391 22604 server_base.cc:1034] running on GCE node
W20260928 02:45:26.864146 22610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:26.864820 22604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:26.865463 22604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:26.866654 22604 hybrid_clock.cc:648] HybridClock initialized: now 1790563526866635 us; error 28 us; skew 500 ppm
I20260928 02:45:26.868520 22604 webserver.cc:460] Webserver started at http://127.2.123.3:36669/ using document root <none> and password file <none>
I20260928 02:45:26.869040 22604 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:26.869096 22604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:26.869263 22604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:26.871205 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/data/instance:
uuid: "a2fc969800e64018844e456ec5befa5f"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "78e5061280925d740c7dbbc64722b366"
server_key_iv: "8638ef800da181a7c9d7a989cdbfc38d"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.871688 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance:
uuid: "a2fc969800e64018844e456ec5befa5f"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "78e5061280925d740c7dbbc64722b366"
server_key_iv: "8638ef800da181a7c9d7a989cdbfc38d"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.875195 22604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:45:26.877471 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.878664 22604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:26.878764 22604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/wal
uuid: "a2fc969800e64018844e456ec5befa5f"
format_stamp: "Formatted at 2026-09-28 02:45:26 on dist-test-slave-rqvb"
server_key: "78e5061280925d740c7dbbc64722b366"
server_key_iv: "8638ef800da181a7c9d7a989cdbfc38d"
server_key_version: "encryptionkey@0"
I20260928 02:45:26.878858 22604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:26.890038 22604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:26.890650 22604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:26.890846 22604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:26.891377 22604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:26.892233 22604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:26.892277 22604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.892314 22604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:26.892343 22604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:26.900645 22604 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.3:33897
I20260928 02:45:26.900727 22731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.3:33897 every 8 connection(s)
I20260928 02:45:26.901510 22604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/data/info.pb
I20260928 02:45:26.905877 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 22604
I20260928 02:45:26.906005 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/wal/instance
I20260928 02:45:26.906286 2540 external_mini_cluster.cc:1428] Setting key 52cf2c38aab8775e265791ec6d08994c
I20260928 02:45:26.910387 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.903268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.3:48333 (local address 127.2.123.62:33187)
0928 02:45:26.903407 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.903411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.904433 (+ 1022us) server_negotiation.cc:407] Connection header received
0928 02:45:26.905777 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.905780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.905836 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.905912 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:26.906857 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.907398 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.908539 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.908769 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.908857 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.909894 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:26.909898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:26.909901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:26.909913 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:26.909935 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:26.909944 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:26.910044 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.910178 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.910240 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:45:26.911190 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.903631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:33187 (local address 127.2.123.3:48333)
0928 02:45:26.904293 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:26.904332 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:26.905560 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:26.906124 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:26.906133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:26.906226 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:26.906720 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.906733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.907516 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.907519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.908431 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.908441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.908910 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.908915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.909065 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.909651 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:45:26.909653 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:26.909693 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:45:26.909756 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:26.910045 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:26.910054 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:26.910060 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.910174 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.910450 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260928 02:45:26.912170 22732 heartbeater.cc:347] Connected to a master server at 127.2.123.62:33187
I20260928 02:45:26.912389 22732 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:26.912866 22732 heartbeater.cc:511] Master 127.2.123.62:33187 requested a full tablet report, sending...
I20260928 02:45:26.913805 22282 ts_manager.cc:194] Registered new tserver with Master: a2fc969800e64018844e456ec5befa5f (127.2.123.3:33897)
I20260928 02:45:26.914417 22282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.123.3:48333
I20260928 02:45:26.919661 2540 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:45:26.929248 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.921510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.2.123.62:33187)
0928 02:45:26.921633 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.921637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.921768 (+ 131us) server_negotiation.cc:407] Connection header received
0928 02:45:26.921861 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.921864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.921905 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.922001 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:26.922212 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.922740 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.923553 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.923758 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.923831 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.928896 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:26.928900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:26.928902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:26.928914 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:26.928934 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:26.928942 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:26.928997 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.929056 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.929109 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:45:26.931921 22282 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:45:26.934939 22282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:45:26.949397 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.944165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.2.123.1:43995)
0928 02:45:26.944634 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.944646 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.944670 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:45:26.944745 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.944751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.944941 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.945133 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:26.945423 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.946657 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.948048 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.948758 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.948851 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.949033 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.949138 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.949235 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":145,"threads_started":1}
I20260928 02:45:26.949394 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.943481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:43995 (local address 127.0.0.1:44080)
0928 02:45:26.944272 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:26.944333 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:26.944456 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:26.945132 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:26.945138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:26.945157 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:26.945293 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.945301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.946798 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.946801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.947940 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.947948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.948858 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.948863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.948969 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.949006 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.949112 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.949225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":13,"thread_start_us":113,"threads_started":1}
I20260928 02:45:26.949700 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.944350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.2.123.3:33897)
0928 02:45:26.944737 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.944747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.944951 (+ 204us) server_negotiation.cc:407] Connection header received
0928 02:45:26.945047 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.945051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.945208 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.945341 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:26.945566 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.946328 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.947877 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.948745 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.948863 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.949033 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.949456 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.949553 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":114,"threads_started":1}
I20260928 02:45:26.949925 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.944180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:33897 (local address 127.0.0.1:59324)
0928 02:45:26.944857 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:26.944868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:26.944950 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:26.945328 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:26.945331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:26.945347 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:26.945466 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.945472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.946432 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.946436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.947755 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.947765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.949111 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.949115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.949212 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.949233 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.949300 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.949365 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260928 02:45:26.950158 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.943341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.2:43689 (local address 127.0.0.1:36388)
0928 02:45:26.944575 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:26.944589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:26.944683 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:26.945301 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:26.945304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:26.945321 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:26.945484 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.945491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.946832 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.946834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.948580 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:26.948588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.949762 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.949765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:26.949857 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.949890 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:45:26.949954 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:45:26.950040 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":69,"thread_start_us":88,"threads_started":1}
I20260928 02:45:26.950649 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:26.943450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.2.123.2:43689)
0928 02:45:26.944671 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:26.944684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:26.944709 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:45:26.944798 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:26.944803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:26.945002 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:26.945144 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:26.945602 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.946730 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.948693 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:26.949434 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:26.949531 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:26.949704 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:26.950404 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 02:45:26.950502 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":137,"threads_started":1}
I20260928 02:45:26.952644 22666 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9b44195e0e4461b61cbcc3758b8cba (DEFAULT_TABLE table=test-table [id=ed5422581f2045809da019f9c049f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:26.952839 22535 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9b44195e0e4461b61cbcc3758b8cba (DEFAULT_TABLE table=test-table [id=ed5422581f2045809da019f9c049f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:26.952795 22404 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9b44195e0e4461b61cbcc3758b8cba (DEFAULT_TABLE table=test-table [id=ed5422581f2045809da019f9c049f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:26.953598 22666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9b44195e0e4461b61cbcc3758b8cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:26.953783 22535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9b44195e0e4461b61cbcc3758b8cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:26.953783 22404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9b44195e0e4461b61cbcc3758b8cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:26.959432 22749 tablet_bootstrap.cc:492] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Bootstrap starting.
I20260928 02:45:26.960822 22751 tablet_bootstrap.cc:492] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e: Bootstrap starting.
I20260928 02:45:26.961269 22750 tablet_bootstrap.cc:492] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Bootstrap starting.
I20260928 02:45:26.961833 22749 tablet_bootstrap.cc:654] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:26.962700 22749 log.cc:824] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:26.962894 22751 tablet_bootstrap.cc:654] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:26.962955 22750 tablet_bootstrap.cc:654] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:26.963610 22751 log.cc:824] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:26.963671 22750 log.cc:824] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:26.964674 22749 tablet_bootstrap.cc:492] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: No bootstrap required, opened a new log
I20260928 02:45:26.964857 22749 ts_tablet_manager.cc:1397] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260928 02:45:26.965541 22750 tablet_bootstrap.cc:492] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: No bootstrap required, opened a new log
I20260928 02:45:26.965735 22750 ts_tablet_manager.cc:1397] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:45:26.966972 22751 tablet_bootstrap.cc:492] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e: No bootstrap required, opened a new log
I20260928 02:45:26.967145 22751 ts_tablet_manager.cc:1397] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 02:45:26.969082 22749 raft_consensus.cc:348] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:26.969357 22749 raft_consensus.cc:374] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:26.969414 22749 raft_consensus.cc:729] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fc969800e64018844e456ec5befa5f, State: Initialized, Role: FOLLOWER
I20260928 02:45:26.969870 22749 consensus_queue.cc:260] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:26.969789 22750 raft_consensus.cc:348] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:26.970044 22750 raft_consensus.cc:374] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:26.970096 22750 raft_consensus.cc:729] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3bb522c49c442af8346bb494cc92366, State: Initialized, Role: FOLLOWER
I20260928 02:45:26.970206 22751 raft_consensus.cc:348] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:26.970450 22751 raft_consensus.cc:374] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:26.970505 22751 raft_consensus.cc:729] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20be496ba24466b803637a2ac753d8e, State: Initialized, Role: FOLLOWER
I20260928 02:45:26.970539 22750 consensus_queue.cc:260] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:26.970646 22732 heartbeater.cc:503] Master 127.2.123.62:33187 was elected leader, sending a full tablet report...
I20260928 02:45:26.970911 22749 ts_tablet_manager.cc:1428] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 02:45:26.970901 22751 consensus_queue.cc:260] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:26.971413 22601 heartbeater.cc:503] Master 127.2.123.62:33187 was elected leader, sending a full tablet report...
I20260928 02:45:26.971766 22751 ts_tablet_manager.cc:1428] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:45:26.972076 22470 heartbeater.cc:503] Master 127.2.123.62:33187 was elected leader, sending a full tablet report...
I20260928 02:45:26.972391 22750 ts_tablet_manager.cc:1428] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
W20260928 02:45:27.032658 22471 tablet.cc:2369] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:45:27.152460 22733 tablet.cc:2369] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:27.171396 22756 raft_consensus.cc:482] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:27.171613 22756 raft_consensus.cc:504] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:27.172696 22756 leader_election.cc:288] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20be496ba24466b803637a2ac753d8e (127.2.123.1:43995), a2fc969800e64018844e456ec5befa5f (127.2.123.3:33897)
I20260928 02:45:27.176324 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:27.172972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.3:33897 (local address 127.2.123.2:44677)
0928 02:45:27.173098 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:27.173113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:27.173209 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:27.173443 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:27.173446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:27.173464 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:27.173570 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:27.173576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.174315 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.174318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:27.175093 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:27.175101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.175964 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.175967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:27.176046 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:27.176065 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:45:27.176130 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:45:27.176197 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:45:27.176795 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:27.173089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:44677 (local address 127.2.123.3:33897)
0928 02:45:27.173209 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:27.173212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:27.173225 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:45:27.173296 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:27.173298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:27.173337 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:27.173410 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:27.173680 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.174217 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.175215 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.175732 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.175829 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:27.175900 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:27.176612 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 02:45:27.176685 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:45:27.176808 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:27.173058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.123.2:57063 (local address 127.2.123.1:43995)
0928 02:45:27.173172 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:27.173177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:27.173721 (+ 544us) server_negotiation.cc:407] Connection header received
0928 02:45:27.173787 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:27.173790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:27.173834 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:27.173893 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:27.174171 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.174667 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.175684 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.176208 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.176299 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:27.176353 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:27.176631 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:45:27.176709 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:45:27.176805 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:27.172977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:43995 (local address 127.2.123.2:57063)
0928 02:45:27.173607 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:27.173617 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:27.173706 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:27.173914 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:27.173917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:27.173931 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:27.174065 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:27.174070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.174777 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.174780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:27.175575 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:27.175583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.176324 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.176328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:27.176512 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:27.176536 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:45:27.176602 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:45:27.176668 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":296,"thread_start_us":114,"threads_started":1}
I20260928 02:45:27.177366 22685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9b44195e0e4461b61cbcc3758b8cba" candidate_uuid: "b3bb522c49c442af8346bb494cc92366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fc969800e64018844e456ec5befa5f" is_pre_election: true
I20260928 02:45:27.177525 22424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9b44195e0e4461b61cbcc3758b8cba" candidate_uuid: "b3bb522c49c442af8346bb494cc92366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20be496ba24466b803637a2ac753d8e" is_pre_election: true
I20260928 02:45:27.177655 22685 raft_consensus.cc:2454] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3bb522c49c442af8346bb494cc92366 in term 0.
I20260928 02:45:27.177810 22424 raft_consensus.cc:2454] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3bb522c49c442af8346bb494cc92366 in term 0.
I20260928 02:45:27.178161 22491 leader_election.cc:302] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2fc969800e64018844e456ec5befa5f, b3bb522c49c442af8346bb494cc92366; no voters:
I20260928 02:45:27.178406 22756 raft_consensus.cc:2781] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:45:27.178491 22756 raft_consensus.cc:482] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:45:27.178539 22756 raft_consensus.cc:3037] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:27.179499 22756 raft_consensus.cc:504] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:27.179819 22756 leader_election.cc:288] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [CANDIDATE]: Term 1 election: Requested vote from peers f20be496ba24466b803637a2ac753d8e (127.2.123.1:43995), a2fc969800e64018844e456ec5befa5f (127.2.123.3:33897)
I20260928 02:45:27.180136 22685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9b44195e0e4461b61cbcc3758b8cba" candidate_uuid: "b3bb522c49c442af8346bb494cc92366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fc969800e64018844e456ec5befa5f"
I20260928 02:45:27.180143 22424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9b44195e0e4461b61cbcc3758b8cba" candidate_uuid: "b3bb522c49c442af8346bb494cc92366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20be496ba24466b803637a2ac753d8e"
I20260928 02:45:27.180240 22685 raft_consensus.cc:3037] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:27.180255 22424 raft_consensus.cc:3037] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:27.181282 22685 raft_consensus.cc:2454] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3bb522c49c442af8346bb494cc92366 in term 1.
I20260928 02:45:27.181348 22424 raft_consensus.cc:2454] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3bb522c49c442af8346bb494cc92366 in term 1.
I20260928 02:45:27.181568 22491 leader_election.cc:302] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2fc969800e64018844e456ec5befa5f, b3bb522c49c442af8346bb494cc92366; no voters:
I20260928 02:45:27.181744 22756 raft_consensus.cc:2781] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:27.181996 22756 raft_consensus.cc:686] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 1 LEADER]: Becoming Leader. State: Replica: b3bb522c49c442af8346bb494cc92366, State: Running, Role: LEADER
I20260928 02:45:27.182358 22756 consensus_queue.cc:237] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } }
I20260928 02:45:27.185390 22277 catalog_manager.cc:5495] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3bb522c49c442af8346bb494cc92366 (127.2.123.2). New cstate: current_term: 1 leader_uuid: "b3bb522c49c442af8346bb494cc92366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bb522c49c442af8346bb494cc92366" member_type: VOTER last_known_addr { host: "127.2.123.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 } health_report { overall_health: UNKNOWN } } }
W20260928 02:45:27.216132 22602 tablet.cc:2369] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:27.241626 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:27.238302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.2.123.2:43689)
0928 02:45:27.238438 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:27.238442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:27.238457 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:45:27.238568 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:27.238570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:27.238631 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:27.238735 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:27.238957 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.239481 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.240300 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:27.240530 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:27.240621 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:27.240839 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:27.241272 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:27.241371 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:27.241410 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 02:45:27.241458 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:45:27.247821 22685 raft_consensus.cc:1261] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 1 FOLLOWER]: Refusing update from remote peer b3bb522c49c442af8346bb494cc92366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:27.247815 22424 raft_consensus.cc:1261] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 1 FOLLOWER]: Refusing update from remote peer b3bb522c49c442af8346bb494cc92366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:27.248535 22759 consensus_queue.cc:1035] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20be496ba24466b803637a2ac753d8e" member_type: VOTER last_known_addr { host: "127.2.123.1" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:27.248728 22756 consensus_queue.cc:1035] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2fc969800e64018844e456ec5befa5f" member_type: VOTER last_known_addr { host: "127.2.123.3" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:27.265602 22282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:27.265777 22282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:27.268142 22282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6e54a0403a24ae89ec1246f243c4cf1: Sending DeleteTablet for 3 replicas of tablet 3d9b44195e0e4461b61cbcc3758b8cba
I20260928 02:45:27.268939 22535 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9b44195e0e4461b61cbcc3758b8cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:27 UTC) from {username='slave'} at 127.0.0.1:36388
I20260928 02:45:27.269018 22404 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9b44195e0e4461b61cbcc3758b8cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:27 UTC) from {username='slave'} at 127.0.0.1:44080
I20260928 02:45:27.269109 22666 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9b44195e0e4461b61cbcc3758b8cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:27 UTC) from {username='slave'} at 127.0.0.1:59324
I20260928 02:45:27.269516 22773 tablet_replica.cc:330] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e: stopping tablet replica
I20260928 02:45:27.269747 22772 tablet_replica.cc:330] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: stopping tablet replica
I20260928 02:45:27.269847 22773 raft_consensus.cc:2229] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:27.270172 22772 raft_consensus.cc:2229] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:45:27.270216 22773 raft_consensus.cc:2258] T 3d9b44195e0e4461b61cbcc3758b8cba P f20be496ba24466b803637a2ac753d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:27.270239 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 22342
I20260928 02:45:27.270653 22772 raft_consensus.cc:2258] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:27.271234 22774 tablet_replica.cc:330] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: stopping tablet replica
I20260928 02:45:27.271508 22774 raft_consensus.cc:2229] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:27.271786 22774 raft_consensus.cc:2258] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:27.273259 22774 ts_tablet_manager.cc:1905] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:27.274375 22772 ts_tablet_manager.cc:1905] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:27.276194 22774 ts_tablet_manager.cc:1918] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:27.276288 22774 log.cc:1196] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-2/wal/wals/3d9b44195e0e4461b61cbcc3758b8cba
I20260928 02:45:27.276546 22774 ts_tablet_manager.cc:1939] T 3d9b44195e0e4461b61cbcc3758b8cba P a2fc969800e64018844e456ec5befa5f: Deleting consensus metadata
I20260928 02:45:27.277257 22772 ts_tablet_manager.cc:1918] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:27.277413 22772 log.cc:1196] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Deleting WAL directory at /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563278472902-2540-0/minicluster-data/ts-1/wal/wals/3d9b44195e0e4461b61cbcc3758b8cba
W20260928 02:45:27.277581 22268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.123.1:43995 (error 108)
I20260928 02:45:27.277814 22772 ts_tablet_manager.cc:1939] T 3d9b44195e0e4461b61cbcc3758b8cba P b3bb522c49c442af8346bb494cc92366: Deleting consensus metadata
I20260928 02:45:27.278227 22269 catalog_manager.cc:4841] TS a2fc969800e64018844e456ec5befa5f (127.2.123.3:33897): tablet 3d9b44195e0e4461b61cbcc3758b8cba (table test-table [id=ed5422581f2045809da019f9c049f618]) successfully deleted
W20260928 02:45:27.278834 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:27.278543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.1:43995 (local address 127.0.0.1:44088)
0928 02:45:27.278671 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:27.278734 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.1:43995: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:45:27.278957 22268 catalog_manager.cc:4584] TS f20be496ba24466b803637a2ac753d8e (127.2.123.1:43995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d9b44195e0e4461b61cbcc3758b8cba: Network error: Client connection negotiation failed: client connection to 127.2.123.1:43995: connect: Connection reset by peer (error 104)
I20260928 02:45:27.279096 22267 catalog_manager.cc:4841] TS b3bb522c49c442af8346bb494cc92366 (127.2.123.2:43689): tablet 3d9b44195e0e4461b61cbcc3758b8cba (table test-table [id=ed5422581f2045809da019f9c049f618]) successfully deleted
I20260928 02:45:27.279227 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 22473
I20260928 02:45:27.284138 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 22604
I20260928 02:45:27.288755 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 22251
2026-09-28T02:45:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1180 ms)
[----------] 6 tests from EncryptionPolicyTest (9985 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:27 dist-test-slave-rqvb krb5kdc[22778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:27 dist-test-slave-rqvb krb5kdc[22778](info): set up 2 sockets
Sep 28 02:45:27 dist-test-slave-rqvb krb5kdc[22778](info): commencing operation
krb5kdc: starting...
W20260928 02:45:28.241840 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:28 dist-test-slave-rqvb krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:28.360458 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:28.467298 22791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:28.467567 22791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:28.467625 22791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:28.471560 22791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:28.471614 22791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:28.471640 22791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:28.471661 22791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:28.471688 22791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:28.476727 22791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:28.477690 22791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:28.478699 22791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:28.486655 22799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:28.486632 22797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:28.486677 22796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:28.486893 22791 server_base.cc:1034] running on GCE node
I20260928 02:45:28.487483 22791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:28.488610 22791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:28.489784 22791 hybrid_clock.cc:648] HybridClock initialized: now 1790563528489760 us; error 41 us; skew 500 ppm
Sep 28 02:45:28 dist-test-slave-rqvb krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:28.492431 22791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:28.493261 22791 webserver.cc:460] Webserver started at http://127.0.0.1:39379/ using document root <none> and password file <none>
I20260928 02:45:28.493775 22791 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:28.493839 22791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:28.494074 22791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:28.496047 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "ac96e569ad6d496f83553c09518f45cd"
format_stamp: "Formatted at 2026-09-28 02:45:28 on dist-test-slave-rqvb"
server_key: "eb6c49d770b817704f5ce5b13757b2ee"
server_key_iv: "56315ddf062a3481280d6c87a4c30354"
server_key_version: "encryptionkey@0"
I20260928 02:45:28.496570 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "ac96e569ad6d496f83553c09518f45cd"
format_stamp: "Formatted at 2026-09-28 02:45:28 on dist-test-slave-rqvb"
server_key: "eb6c49d770b817704f5ce5b13757b2ee"
server_key_iv: "56315ddf062a3481280d6c87a4c30354"
server_key_version: "encryptionkey@0"
I20260928 02:45:28.500082 22791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:45:28.502444 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:28.503630 22791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:28.503744 22791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "ac96e569ad6d496f83553c09518f45cd"
format_stamp: "Formatted at 2026-09-28 02:45:28 on dist-test-slave-rqvb"
server_key: "eb6c49d770b817704f5ce5b13757b2ee"
server_key_iv: "56315ddf062a3481280d6c87a4c30354"
server_key_version: "encryptionkey@0"
I20260928 02:45:28.503836 22791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:28.522962 22791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:28.525707 22791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:28.525890 22791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:28.532652 22791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38491
I20260928 02:45:28.532719 22858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38491 every 8 connection(s)
I20260928 02:45:28.533592 22791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:28.536437 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:28.536563 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 22791
I20260928 02:45:28.536696 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:28.537024 2540 external_mini_cluster.cc:1428] Setting key c14663fd5a923d5a6576cf9b1d7d98c4
I20260928 02:45:28.542698 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd: Bootstrap starting.
I20260928 02:45:28.545020 22859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd: Neither blocks nor log segments found. Creating new log.
Sep 28 02:45:28 dist-test-slave-rqvb krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:45:28.545804 22859 log.cc:824] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:28.547569 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd: No bootstrap required, opened a new log
I20260928 02:45:28.550513 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:28.538508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59591 (local address 127.0.0.1:38491)
0928 02:45:28.538961 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:28.538976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:28.539013 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:45:28.539714 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:28.539731 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:28.540070 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:28.540377 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:28.541178 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:28.542186 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:28.543057 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:28.543520 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:28.543659 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:28.546187 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:28.546192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:28.546205 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:28.546233 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:28.547779 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:28.548250 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:28.548255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:28.548259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:28.548365 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:28.548678 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:28.548681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:28.548682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:28.549064 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:28.549248 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:28.549550 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 02:45:28.549751 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":130,"threads_started":1}
I20260928 02:45:28.551098 22859 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96e569ad6d496f83553c09518f45cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260928 02:45:28.551324 22859 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:28.551383 22859 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac96e569ad6d496f83553c09518f45cd, State: Initialized, Role: FOLLOWER
I20260928 02:45:28.551810 22859 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96e569ad6d496f83553c09518f45cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260928 02:45:28.551939 22859 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:28.552002 22859 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:28.552098 22859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:28.553048 22859 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96e569ad6d496f83553c09518f45cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260928 02:45:28.553289 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:28.553355 22859 leader_election.cc:302] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac96e569ad6d496f83553c09518f45cd; no voters:
I20260928 02:45:28.553649 22859 leader_election.cc:288] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:28.553754 22864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:28.553979 22864 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [term 1 LEADER]: Becoming Leader. State: Replica: ac96e569ad6d496f83553c09518f45cd, State: Running, Role: LEADER
I20260928 02:45:28.554304 22864 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96e569ad6d496f83553c09518f45cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260928 02:45:28.554911 22859 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:28.555691 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac96e569ad6d496f83553c09518f45cd. Latest consensus state: current_term: 1 leader_uuid: "ac96e569ad6d496f83553c09518f45cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96e569ad6d496f83553c09518f45cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } } }
I20260928 02:45:28.555827 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:28.555932 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac96e569ad6d496f83553c09518f45cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac96e569ad6d496f83553c09518f45cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } } }
I20260928 02:45:28.556020 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:28.556291 22878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:28.559509 22878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:28.564461 22878 catalog_manager.cc:1347] Generated new cluster ID: c7b933a707a94f7f9227a2eb6198e5af
I20260928 02:45:28.564517 22878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:28.566099 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:28.555440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38013 (local address 127.0.0.1:38491)
0928 02:45:28.555543 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:28.555547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:28.556272 (+ 725us) server_negotiation.cc:407] Connection header received
0928 02:45:28.556451 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:28.556454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:28.556499 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:28.556554 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:28.557431 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:28.558188 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:28.559308 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:28.559622 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:28.559694 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:28.561720 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:28.561724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:28.561727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:28.561753 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:28.563967 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:28.564542 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:28.564546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:28.564548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:28.564631 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:28.565029 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:28.565033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:28.565034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:28.565244 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:28.565370 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:28.565781 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 02:45:28.565880 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:45:28.573135 22878 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:28.574311 22878 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:28.583231 22878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac96e569ad6d496f83553c09518f45cd: Generated new TSK 0
I20260928 02:45:28.583884 22878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:28.587637 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 22791
W20260928 02:45:28.591930 22776 connection.cc:535] client connection to 127.0.0.1:38491 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38491): Connection reset by peer (error 104)
2026-09-28T02:45:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:28 dist-test-slave-rqvb krb5kdc[22890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:28 dist-test-slave-rqvb krb5kdc[22890](info): set up 2 sockets
Sep 28 02:45:28 dist-test-slave-rqvb krb5kdc[22890](info): commencing operation
krb5kdc: starting...
W20260928 02:45:29.538251 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:29 dist-test-slave-rqvb krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:29.654814 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:29.761281 22903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:29.761561 22903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:29.761605 22903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:29.765563 22903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:29.765637 22903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:29.765655 22903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:29.765671 22903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:29.765686 22903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:29.771284 22903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:29.772353 22903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:29.773365 22903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:29.781574 22911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:29.781625 22909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:29.781603 22908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:29.781946 22903 server_base.cc:1034] running on GCE node
I20260928 02:45:29.782451 22903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:29.783494 22903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:29.784674 22903 hybrid_clock.cc:648] HybridClock initialized: now 1790563529784652 us; error 31 us; skew 500 ppm
Sep 28 02:45:29 dist-test-slave-rqvb krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:29.787264 22903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:29.788074 22903 webserver.cc:460] Webserver started at http://127.0.0.1:38211/ using document root <none> and password file <none>
I20260928 02:45:29.788611 22903 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:29.788683 22903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:29.788884 22903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:29.790925 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "987df0e774964b6db876ab0cd44c0f13"
format_stamp: "Formatted at 2026-09-28 02:45:29 on dist-test-slave-rqvb"
server_key: "1064d6a14ab87d7eb977074e010c9c3d"
server_key_iv: "653a8801a62e6932624970fac6436709"
server_key_version: "encryptionkey@0"
I20260928 02:45:29.791438 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "987df0e774964b6db876ab0cd44c0f13"
format_stamp: "Formatted at 2026-09-28 02:45:29 on dist-test-slave-rqvb"
server_key: "1064d6a14ab87d7eb977074e010c9c3d"
server_key_iv: "653a8801a62e6932624970fac6436709"
server_key_version: "encryptionkey@0"
I20260928 02:45:29.794955 22903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:45:29.797219 22918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:29.798440 22903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:29.798557 22903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "987df0e774964b6db876ab0cd44c0f13"
format_stamp: "Formatted at 2026-09-28 02:45:29 on dist-test-slave-rqvb"
server_key: "1064d6a14ab87d7eb977074e010c9c3d"
server_key_iv: "653a8801a62e6932624970fac6436709"
server_key_version: "encryptionkey@0"
I20260928 02:45:29.798655 22903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:29.822474 22903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:29.825876 22903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:29.826108 22903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:29.833624 22903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45135
I20260928 02:45:29.833690 22970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45135 every 8 connection(s)
I20260928 02:45:29.834631 22903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:29.837466 22971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:29.840637 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 22903
I20260928 02:45:29.840785 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:29.841092 2540 external_mini_cluster.cc:1428] Setting key 3a4efc8b60925754935d2d642b26b617
I20260928 02:45:29.843673 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13: Bootstrap starting.
I20260928 02:45:29.846109 22971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:29.847024 22971 log.cc:824] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:29.849236 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13: No bootstrap required, opened a new log
I20260928 02:45:29.850436 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:29.842612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46101 (local address 127.0.0.1:45135)
0928 02:45:29.843027 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:29.843043 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:29.843080 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:45:29.843722 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:29.843742 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:29.844132 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:29.844435 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:29.844775 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:29.845753 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:29.846988 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:29.847599 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:29.847786 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:29.848688 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:29.848693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:29.848707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:29.849091 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:29.849151 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:29.849174 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:29.849274 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:29.849486 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:45:29.849623 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":109,"threads_started":1}
I20260928 02:45:29.852376 22971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987df0e774964b6db876ab0cd44c0f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260928 02:45:29.852620 22971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:29.852687 22971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987df0e774964b6db876ab0cd44c0f13, State: Initialized, Role: FOLLOWER
I20260928 02:45:29.853116 22971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987df0e774964b6db876ab0cd44c0f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260928 02:45:29.853250 22971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:29.853353 22971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:29.853494 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:29.853489 22971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:29.854646 22971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987df0e774964b6db876ab0cd44c0f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260928 02:45:29.855042 22971 leader_election.cc:302] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 987df0e774964b6db876ab0cd44c0f13; no voters:
I20260928 02:45:29.855412 22971 leader_election.cc:288] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:29.855473 22980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:29.855758 22980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [term 1 LEADER]: Becoming Leader. State: Replica: 987df0e774964b6db876ab0cd44c0f13, State: Running, Role: LEADER
I20260928 02:45:29.856242 22980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987df0e774964b6db876ab0cd44c0f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260928 02:45:29.856745 22971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:29.858014 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "987df0e774964b6db876ab0cd44c0f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987df0e774964b6db876ab0cd44c0f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } } }
I20260928 02:45:29.858141 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:29.858414 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987df0e774964b6db876ab0cd44c0f13. Latest consensus state: current_term: 1 leader_uuid: "987df0e774964b6db876ab0cd44c0f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987df0e774964b6db876ab0cd44c0f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } } }
I20260928 02:45:29.858496 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:29.860708 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:29.855720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60779 (local address 127.0.0.1:45135)
0928 02:45:29.855827 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:29.855831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:29.856095 (+ 264us) server_negotiation.cc:407] Connection header received
0928 02:45:29.856297 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:29.856300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:29.856350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:29.856461 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:29.856723 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:29.857525 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:29.858701 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:29.859012 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:29.859104 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:29.860141 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:29.860145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:29.860149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:29.860162 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:29.860191 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:29.860203 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:29.860304 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:29.860536 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:45:29.860574 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 02:45:29.861657 22996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:29.861732 22996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:29.861799 22991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:29.862548 22991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:29.868064 22991 catalog_manager.cc:1347] Generated new cluster ID: 16d3b4f14a19410ab8005f47df7b00fd
I20260928 02:45:29.868149 22991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:29.882485 22991 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:29.883814 22991 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:29.890129 22991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 987df0e774964b6db876ab0cd44c0f13: Generated new TSK 0
I20260928 02:45:29.890797 22991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:29.895376 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 22903
W20260928 02:45:29.899720 22888 connection.cc:535] client connection to 127.0.0.1:45135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45135): Connection reset by peer (error 104)
2026-09-28T02:45:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:29 dist-test-slave-rqvb krb5kdc[23002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:29 dist-test-slave-rqvb krb5kdc[23002](info): set up 2 sockets
Sep 28 02:45:29 dist-test-slave-rqvb krb5kdc[23002](info): commencing operation
krb5kdc: starting...
W20260928 02:45:30.850013 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:30 dist-test-slave-rqvb krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:30.968745 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:31.075743 23015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:31.076023 23015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:31.076064 23015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:31.080047 23015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:31.080109 23015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:31.080127 23015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:31.080142 23015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:31.080156 23015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:31.085505 23015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:31.086637 23015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:31.087620 23015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:31.095860 23023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:31.095860 23020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:31.095880 23021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:31.095907 23015 server_base.cc:1034] running on GCE node
I20260928 02:45:31.096570 23015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:31.097672 23015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:31.098841 23015 hybrid_clock.cc:648] HybridClock initialized: now 1790563531098822 us; error 32 us; skew 500 ppm
Sep 28 02:45:31 dist-test-slave-rqvb krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:31.101432 23015 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:31.102396 23015 webserver.cc:460] Webserver started at http://127.0.0.1:40443/ using document root <none> and password file <none>
I20260928 02:45:31.102938 23015 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:31.103003 23015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:31.103219 23015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:31.105195 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "243e9ebadf7940fabe7f8e94b8f0ca71"
format_stamp: "Formatted at 2026-09-28 02:45:31 on dist-test-slave-rqvb"
server_key: "e9584d53f9c5ed6d606d37b5eea9ab6e"
server_key_iv: "847e0a0b698be65bf81a7aa1672b3060"
server_key_version: "encryptionkey@0"
I20260928 02:45:31.105754 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "243e9ebadf7940fabe7f8e94b8f0ca71"
format_stamp: "Formatted at 2026-09-28 02:45:31 on dist-test-slave-rqvb"
server_key: "e9584d53f9c5ed6d606d37b5eea9ab6e"
server_key_iv: "847e0a0b698be65bf81a7aa1672b3060"
server_key_version: "encryptionkey@0"
I20260928 02:45:31.109347 23015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:45:31.111791 23030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:31.112998 23015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:45:31.113132 23015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "243e9ebadf7940fabe7f8e94b8f0ca71"
format_stamp: "Formatted at 2026-09-28 02:45:31 on dist-test-slave-rqvb"
server_key: "e9584d53f9c5ed6d606d37b5eea9ab6e"
server_key_iv: "847e0a0b698be65bf81a7aa1672b3060"
server_key_version: "encryptionkey@0"
I20260928 02:45:31.113231 23015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:31.117594 23015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:31.120469 23015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:31.120625 23015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:31.127105 23015 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46333
I20260928 02:45:31.127111 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46333 every 8 connection(s)
I20260928 02:45:31.127967 23015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:31.130581 23083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:31.134519 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23015
I20260928 02:45:31.134670 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:31.134949 2540 external_mini_cluster.cc:1428] Setting key c3726779d3efc7474a471d9fc4838144
I20260928 02:45:31.136221 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71: Bootstrap starting.
I20260928 02:45:31.138480 23083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:31.139307 23083 log.cc:824] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:31.139891 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:31.136260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47475 (local address 127.0.0.1:46333)
0928 02:45:31.136664 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:31.136678 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:31.136713 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:45:31.137360 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:31.137377 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:31.137660 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:31.137926 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:31.138713 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:31.138717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:31.138729 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:31.138739 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:31.138791 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:31.138810 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:31.138892 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:31.139006 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 02:45:31.139176 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260928 02:45:31.141144 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71: No bootstrap required, opened a new log
I20260928 02:45:31.142531 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:31.145531 23083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243e9ebadf7940fabe7f8e94b8f0ca71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260928 02:45:31.145813 23083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:31.145895 23083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243e9ebadf7940fabe7f8e94b8f0ca71, State: Initialized, Role: FOLLOWER
I20260928 02:45:31.146180 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:31.144691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58817 (local address 127.0.0.1:46333)
0928 02:45:31.144837 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:31.144840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:31.144922 (+ 82us) server_negotiation.cc:407] Connection header received
0928 02:45:31.145028 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:31.145030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:31.145056 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:31.145130 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:31.145758 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:31.145760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:31.145762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:31.145771 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:31.145797 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:31.145811 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:31.145892 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:31.146009 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 02:45:31.146047 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:45:31.146401 23083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243e9ebadf7940fabe7f8e94b8f0ca71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260928 02:45:31.146549 23083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:31.146615 23083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:31.146705 23083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:31.147658 23083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243e9ebadf7940fabe7f8e94b8f0ca71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260928 02:45:31.147936 23083 leader_election.cc:302] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 243e9ebadf7940fabe7f8e94b8f0ca71; no voters:
I20260928 02:45:31.148175 23083 leader_election.cc:288] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:31.148280 23093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:31.148496 23093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [term 1 LEADER]: Becoming Leader. State: Replica: 243e9ebadf7940fabe7f8e94b8f0ca71, State: Running, Role: LEADER
I20260928 02:45:31.148877 23093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243e9ebadf7940fabe7f8e94b8f0ca71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260928 02:45:31.149263 23083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:31.150385 23095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 243e9ebadf7940fabe7f8e94b8f0ca71. Latest consensus state: current_term: 1 leader_uuid: "243e9ebadf7940fabe7f8e94b8f0ca71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243e9ebadf7940fabe7f8e94b8f0ca71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } } }
I20260928 02:45:31.150501 23095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:31.150789 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "243e9ebadf7940fabe7f8e94b8f0ca71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243e9ebadf7940fabe7f8e94b8f0ca71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } } }
I20260928 02:45:31.150862 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71 [sys.catalog]: This master's current role is: LEADER
W20260928 02:45:31.153746 23108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:31.153810 23108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:31.153883 23104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:31.154577 23104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:31.159811 23104 catalog_manager.cc:1347] Generated new cluster ID: 99d22f61542a4f698c5e544f9f258165
I20260928 02:45:31.159871 23104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:31.186601 23104 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:31.186937 23104 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:31.196404 23104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 243e9ebadf7940fabe7f8e94b8f0ca71: Generated new TSK 0
I20260928 02:45:31.197136 23104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:31.200417 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23015
2026-09-28T02:45:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:31 dist-test-slave-rqvb krb5kdc[23114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:31 dist-test-slave-rqvb krb5kdc[23114](info): set up 2 sockets
Sep 28 02:45:31 dist-test-slave-rqvb krb5kdc[23114](info): commencing operation
krb5kdc: starting...
W20260928 02:45:32.149935 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:32 dist-test-slave-rqvb krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:32.265177 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:32.371644 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:32.371918 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:32.371977 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:32.375970 23127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:32.376031 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:32.376058 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:32.376080 23127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:32.376108 23127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:32.381269 23127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:32.382355 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:32.383343 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:32.391458 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:32.391532 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:32.391453 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:32.391703 23127 server_base.cc:1034] running on GCE node
I20260928 02:45:32.392269 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:32.393302 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:32.394454 23127 hybrid_clock.cc:648] HybridClock initialized: now 1790563532394438 us; error 34 us; skew 500 ppm
Sep 28 02:45:32 dist-test-slave-rqvb krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:32.399972 23127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:32.400840 23127 webserver.cc:460] Webserver started at http://127.0.0.1:46353/ using document root <none> and password file <none>
I20260928 02:45:32.401365 23127 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:32.401430 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:32.401620 23127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:32.403610 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "db715192612a45bf8762075a5cf3c8cf"
format_stamp: "Formatted at 2026-09-28 02:45:32 on dist-test-slave-rqvb"
server_key: "64f00863331d97df63dddb2a1a0f3fe2"
server_key_iv: "e17b45fc67b674497df16157e0b5409f"
server_key_version: "encryptionkey@0"
I20260928 02:45:32.404122 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "db715192612a45bf8762075a5cf3c8cf"
format_stamp: "Formatted at 2026-09-28 02:45:32 on dist-test-slave-rqvb"
server_key: "64f00863331d97df63dddb2a1a0f3fe2"
server_key_iv: "e17b45fc67b674497df16157e0b5409f"
server_key_version: "encryptionkey@0"
I20260928 02:45:32.407625 23127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:45:32.409927 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:32.411120 23127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:32.411218 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "db715192612a45bf8762075a5cf3c8cf"
format_stamp: "Formatted at 2026-09-28 02:45:32 on dist-test-slave-rqvb"
server_key: "64f00863331d97df63dddb2a1a0f3fe2"
server_key_iv: "e17b45fc67b674497df16157e0b5409f"
server_key_version: "encryptionkey@0"
I20260928 02:45:32.411303 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:32.427289 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:32.430202 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:32.430353 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:32.436900 23127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40335
I20260928 02:45:32.436923 23194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40335 every 8 connection(s)
I20260928 02:45:32.437803 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:32.440665 23195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:32.440984 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23127
I20260928 02:45:32.441136 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:32.441442 2540 external_mini_cluster.cc:1428] Setting key 4eda22491937bdf549f7f100302515c8
I20260928 02:45:32.447623 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf: Bootstrap starting.
I20260928 02:45:32.449837 23195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf: Neither blocks nor log segments found. Creating new log.
Sep 28 02:45:32 dist-test-slave-rqvb krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:45:32.450589 23195 log.cc:824] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:32.452929 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf: No bootstrap required, opened a new log
I20260928 02:45:32.455926 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:32.442790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44021 (local address 127.0.0.1:40335)
0928 02:45:32.443251 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:32.443269 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:32.443312 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:45:32.443995 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:32.444017 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:32.444459 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:32.444821 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:32.445486 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:32.446965 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:32.447859 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:32.448313 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:32.448457 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:32.450868 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:32.450873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:32.450885 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:32.450911 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:32.452817 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:32.453276 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:32.453282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:32.453288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:32.453437 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:32.453753 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:32.453756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:32.453758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:32.454281 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:32.454465 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:32.454678 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 02:45:32.454979 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
I20260928 02:45:32.457295 23195 raft_consensus.cc:348] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db715192612a45bf8762075a5cf3c8cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } }
I20260928 02:45:32.457504 23195 raft_consensus.cc:374] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:32.457536 23195 raft_consensus.cc:729] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db715192612a45bf8762075a5cf3c8cf, State: Initialized, Role: FOLLOWER
I20260928 02:45:32.458007 23195 consensus_queue.cc:260] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db715192612a45bf8762075a5cf3c8cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } }
I20260928 02:45:32.458159 23195 raft_consensus.cc:388] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:32.458269 23195 raft_consensus.cc:482] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:32.458367 23195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:32.458817 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:32.459301 23195 raft_consensus.cc:504] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db715192612a45bf8762075a5cf3c8cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } }
I20260928 02:45:32.459601 23195 leader_election.cc:302] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db715192612a45bf8762075a5cf3c8cf; no voters:
I20260928 02:45:32.459885 23195 leader_election.cc:288] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:32.459972 23201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:32.460192 23201 raft_consensus.cc:686] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [term 1 LEADER]: Becoming Leader. State: Replica: db715192612a45bf8762075a5cf3c8cf, State: Running, Role: LEADER
I20260928 02:45:32.460474 23201 consensus_queue.cc:237] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db715192612a45bf8762075a5cf3c8cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } }
I20260928 02:45:32.461158 23195 sys_catalog.cc:564] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:32.462030 23205 sys_catalog.cc:455] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader db715192612a45bf8762075a5cf3c8cf. Latest consensus state: current_term: 1 leader_uuid: "db715192612a45bf8762075a5cf3c8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db715192612a45bf8762075a5cf3c8cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } } }
I20260928 02:45:32.462164 23205 sys_catalog.cc:458] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:32.462531 23214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:32.462478 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db715192612a45bf8762075a5cf3c8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db715192612a45bf8762075a5cf3c8cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40335 } } }
I20260928 02:45:32.465045 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:32.466172 23214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:32.469031 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:32.460987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47301 (local address 127.0.0.1:40335)
0928 02:45:32.461081 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:32.461095 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:32.461106 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:45:32.461137 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:32.461139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:32.461176 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:32.461235 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:32.462038 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:32.462823 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:32.463895 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:32.464214 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:32.464305 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:32.465745 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:32.465749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:32.465751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:32.465776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:32.467474 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:32.467838 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:32.467842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:32.467843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:32.467918 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:32.468160 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:32.468163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:32.468165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:32.468391 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:32.468523 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:32.468762 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:45:32.468879 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:45:32.471846 23214 catalog_manager.cc:1347] Generated new cluster ID: a1c3d6453a8344be9eaf7664ed50e67e
I20260928 02:45:32.471903 23214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:32.484530 23214 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:32.485477 23214 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:32.489096 23214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db715192612a45bf8762075a5cf3c8cf: Generated new TSK 0
I20260928 02:45:32.489727 23214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:32.494809 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23127
2026-09-28T02:45:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:32 dist-test-slave-rqvb krb5kdc[23226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:32 dist-test-slave-rqvb krb5kdc[23226](info): set up 2 sockets
Sep 28 02:45:32 dist-test-slave-rqvb krb5kdc[23226](info): commencing operation
krb5kdc: starting...
W20260928 02:45:33.445844 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:33 dist-test-slave-rqvb krb5kdc[23226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:33.559113 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:33.664696 23239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:33.664956 23239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:33.665011 23239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:33.668924 23239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:33.668988 23239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:33.669014 23239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:33.669034 23239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:33.669055 23239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:33.674264 23239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:33.675285 23239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:33.676270 23239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:33.684576 23244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:33.684572 23245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:33.684652 23247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:33.685035 23239 server_base.cc:1034] running on GCE node
I20260928 02:45:33.685578 23239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:33.686676 23239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:33.687837 23239 hybrid_clock.cc:648] HybridClock initialized: now 1790563533687816 us; error 38 us; skew 500 ppm
Sep 28 02:45:33 dist-test-slave-rqvb krb5kdc[23226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:33.690546 23239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:33.691399 23239 webserver.cc:460] Webserver started at http://127.0.0.1:37729/ using document root <none> and password file <none>
I20260928 02:45:33.691938 23239 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:33.692001 23239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:33.692224 23239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:33.694244 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "2457dd97811945c78db1f9a3a8f4c5e7"
format_stamp: "Formatted at 2026-09-28 02:45:33 on dist-test-slave-rqvb"
server_key: "dfbeb93e14d9ec82e396983214264492"
server_key_iv: "cc0010d9ef532b383bd255156c4806e7"
server_key_version: "encryptionkey@0"
I20260928 02:45:33.694762 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "2457dd97811945c78db1f9a3a8f4c5e7"
format_stamp: "Formatted at 2026-09-28 02:45:33 on dist-test-slave-rqvb"
server_key: "dfbeb93e14d9ec82e396983214264492"
server_key_iv: "cc0010d9ef532b383bd255156c4806e7"
server_key_version: "encryptionkey@0"
I20260928 02:45:33.698304 23239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:45:33.700587 23254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:33.701817 23239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:33.701932 23239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "2457dd97811945c78db1f9a3a8f4c5e7"
format_stamp: "Formatted at 2026-09-28 02:45:33 on dist-test-slave-rqvb"
server_key: "dfbeb93e14d9ec82e396983214264492"
server_key_iv: "cc0010d9ef532b383bd255156c4806e7"
server_key_version: "encryptionkey@0"
I20260928 02:45:33.702075 23239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:33.730477 23239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:33.733783 23239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:33.734020 23239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:33.741353 23239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45979
I20260928 02:45:33.741372 23306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45979 every 8 connection(s)
I20260928 02:45:33.742396 23239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:33.745028 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23239
I20260928 02:45:33.745165 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:33.745349 23307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:33.745483 2540 external_mini_cluster.cc:1428] Setting key f59493143ef3c6a8c9bcb2183e0c6eb8
I20260928 02:45:33.751796 23307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7: Bootstrap starting.
I20260928 02:45:33.754151 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:33.746933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45749 (local address 127.0.0.1:45979)
0928 02:45:33.747347 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:33.747361 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:33.747395 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:45:33.748077 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:33.748095 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:33.748433 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:33.748731 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:33.749160 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:33.750198 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:33.751067 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:33.751530 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:33.751671 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:33.752466 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:33.752472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:33.752484 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:33.752892 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:33.752948 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:33.752970 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:33.753056 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:33.753196 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:45:33.753399 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260928 02:45:33.754392 23307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:33.755125 23307 log.cc:824] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:33.756976 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:33.757313 23307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7: No bootstrap required, opened a new log
I20260928 02:45:33.760689 23307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2457dd97811945c78db1f9a3a8f4c5e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } }
I20260928 02:45:33.760978 23307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:33.761046 23307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2457dd97811945c78db1f9a3a8f4c5e7, State: Initialized, Role: FOLLOWER
I20260928 02:45:33.761528 23307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2457dd97811945c78db1f9a3a8f4c5e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } }
I20260928 02:45:33.761680 23307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:33.761746 23307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:33.761845 23307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:33.763012 23307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2457dd97811945c78db1f9a3a8f4c5e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } }
I20260928 02:45:33.763058 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:33.758837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54231 (local address 127.0.0.1:45979)
0928 02:45:33.758963 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:33.758967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:33.759130 (+ 163us) server_negotiation.cc:407] Connection header received
0928 02:45:33.759238 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:33.759240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:33.759278 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:33.759339 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:33.759615 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:33.760453 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:33.761535 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:33.761790 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:33.761882 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:33.762643 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:33.762646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:33.762648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:33.762659 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:33.762682 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:33.762692 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:33.762746 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:33.762853 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:45:33.762897 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:45:33.763404 23307 leader_election.cc:302] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2457dd97811945c78db1f9a3a8f4c5e7; no voters:
I20260928 02:45:33.763738 23307 leader_election.cc:288] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:33.763847 23317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:33.764055 23317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [term 1 LEADER]: Becoming Leader. State: Replica: 2457dd97811945c78db1f9a3a8f4c5e7, State: Running, Role: LEADER
I20260928 02:45:33.764391 23317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2457dd97811945c78db1f9a3a8f4c5e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } }
I20260928 02:45:33.765061 23307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:33.765559 23317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2457dd97811945c78db1f9a3a8f4c5e7. Latest consensus state: current_term: 1 leader_uuid: "2457dd97811945c78db1f9a3a8f4c5e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2457dd97811945c78db1f9a3a8f4c5e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } } }
I20260928 02:45:33.765668 23317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:33.766275 23318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2457dd97811945c78db1f9a3a8f4c5e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2457dd97811945c78db1f9a3a8f4c5e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45979 } } }
I20260928 02:45:33.766373 23318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:33.766637 23322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:33.769593 23322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:33.774781 23322 catalog_manager.cc:1347] Generated new cluster ID: 616bc10b3954448c9e36e59e2b46b7b2
I20260928 02:45:33.774849 23322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:33.788028 23322 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:33.789275 23322 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:33.795154 23322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2457dd97811945c78db1f9a3a8f4c5e7: Generated new TSK 0
I20260928 02:45:33.796016 23322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:33.803530 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23239
2026-09-28T02:45:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:33 dist-test-slave-rqvb krb5kdc[23338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:33 dist-test-slave-rqvb krb5kdc[23338](info): set up 2 sockets
Sep 28 02:45:33 dist-test-slave-rqvb krb5kdc[23338](info): commencing operation
krb5kdc: starting...
W20260928 02:45:34.758467 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:34 dist-test-slave-rqvb krb5kdc[23338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:34.877470 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:34.985091 23351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:34.985356 23351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:34.985422 23351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:34.989432 23351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:34.989502 23351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:34.989529 23351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:34.989549 23351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:34.989569 23351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:34.994902 23351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:34.995950 23351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:34.996961 23351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:35.005667 23356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:35.005658 23359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:35.005836 23351 server_base.cc:1034] running on GCE node
W20260928 02:45:35.005653 23357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:35.006482 23351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:35.007570 23351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:35.008803 23351 hybrid_clock.cc:648] HybridClock initialized: now 1790563535008786 us; error 27 us; skew 500 ppm
Sep 28 02:45:35 dist-test-slave-rqvb krb5kdc[23338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:35.011602 23351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:35.012537 23351 webserver.cc:460] Webserver started at http://127.0.0.1:33957/ using document root <none> and password file <none>
I20260928 02:45:35.013155 23351 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:35.013238 23351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:35.013474 23351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:35.015653 23351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "c54145787091495498a240586dfb0d77"
format_stamp: "Formatted at 2026-09-28 02:45:35 on dist-test-slave-rqvb"
server_key: "87671f8156c72f5be26275b703351c99"
server_key_iv: "52d9a00d7d3e002736664d5eb81a7a45"
server_key_version: "encryptionkey@0"
I20260928 02:45:35.016218 23351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "c54145787091495498a240586dfb0d77"
format_stamp: "Formatted at 2026-09-28 02:45:35 on dist-test-slave-rqvb"
server_key: "87671f8156c72f5be26275b703351c99"
server_key_iv: "52d9a00d7d3e002736664d5eb81a7a45"
server_key_version: "encryptionkey@0"
I20260928 02:45:35.020012 23351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 02:45:35.022619 23366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:35.023903 23351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:35.024006 23351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "c54145787091495498a240586dfb0d77"
format_stamp: "Formatted at 2026-09-28 02:45:35 on dist-test-slave-rqvb"
server_key: "87671f8156c72f5be26275b703351c99"
server_key_iv: "52d9a00d7d3e002736664d5eb81a7a45"
server_key_version: "encryptionkey@0"
I20260928 02:45:35.024098 23351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:35.028568 23351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:35.031698 23351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:35.031884 23351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:35.038461 23351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43331
I20260928 02:45:35.038512 23418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43331 every 8 connection(s)
I20260928 02:45:35.039393 23351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:35.042403 23419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:35.043355 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23351
I20260928 02:45:35.043488 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:35.043730 2540 external_mini_cluster.cc:1428] Setting key ad4d35ab7ced0571c8485f9d291f36b3
I20260928 02:45:35.049247 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:35.044990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56155 (local address 127.0.0.1:43331)
0928 02:45:35.045438 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:35.045459 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:35.045507 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:45:35.046398 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:35.046423 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:35.046784 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:35.047151 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:35.047816 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:35.047822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:35.047838 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:35.047849 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:35.047910 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:35.047938 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:35.048088 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:35.048196 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:45:35.048364 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260928 02:45:35.049486 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77: Bootstrap starting.
I20260928 02:45:35.051923 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:35.052023 23419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:35.052737 23419 log.cc:824] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:35.054852 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77: No bootstrap required, opened a new log
I20260928 02:45:35.055382 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:35.053800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54557 (local address 127.0.0.1:43331)
0928 02:45:35.053914 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:35.053919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:35.054148 (+ 229us) server_negotiation.cc:407] Connection header received
0928 02:45:35.054232 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:35.054235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:35.054271 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:35.054370 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:35.054998 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:35.055002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:35.055005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:35.055017 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:35.055047 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:35.055061 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:35.055157 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:35.055212 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:45:35.055249 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:45:35.058478 23419 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54145787091495498a240586dfb0d77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260928 02:45:35.058710 23419 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:35.058748 23419 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54145787091495498a240586dfb0d77, State: Initialized, Role: FOLLOWER
I20260928 02:45:35.059125 23419 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54145787091495498a240586dfb0d77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260928 02:45:35.059240 23419 raft_consensus.cc:388] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:35.059288 23419 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:35.059374 23419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:35.060295 23419 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54145787091495498a240586dfb0d77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260928 02:45:35.060642 23419 leader_election.cc:302] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c54145787091495498a240586dfb0d77; no voters:
I20260928 02:45:35.060998 23419 leader_election.cc:288] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:35.061097 23429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:35.061326 23429 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [term 1 LEADER]: Becoming Leader. State: Replica: c54145787091495498a240586dfb0d77, State: Running, Role: LEADER
I20260928 02:45:35.061686 23429 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54145787091495498a240586dfb0d77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260928 02:45:35.062258 23419 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:35.063242 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54145787091495498a240586dfb0d77. Latest consensus state: current_term: 1 leader_uuid: "c54145787091495498a240586dfb0d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54145787091495498a240586dfb0d77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } } }
I20260928 02:45:35.063360 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:35.063705 23438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:35.063663 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54145787091495498a240586dfb0d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54145787091495498a240586dfb0d77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } } }
I20260928 02:45:35.063840 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:35.066500 23438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:35.072310 23438 catalog_manager.cc:1347] Generated new cluster ID: 40d20dcc9b4148f5aac6079c071b6475
I20260928 02:45:35.072386 23438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:35.081238 23438 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:35.081580 23438 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:35.086699 23438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54145787091495498a240586dfb0d77: Generated new TSK 0
I20260928 02:45:35.087347 23438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:35.094596 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23351
2026-09-28T02:45:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:35 dist-test-slave-rqvb krb5kdc[23450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:35 dist-test-slave-rqvb krb5kdc[23450](info): set up 2 sockets
Sep 28 02:45:35 dist-test-slave-rqvb krb5kdc[23450](info): commencing operation
krb5kdc: starting...
W20260928 02:45:36.054004 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:36 dist-test-slave-rqvb krb5kdc[23450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:36.170763 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:36.280094 23463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:36.280382 23463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:36.280441 23463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:36.284490 23463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:36.284554 23463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:36.284579 23463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:36.284602 23463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:36.284631 23463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:36.289896 23463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:36.290982 23463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:36.291989 23463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:36.300359 23469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:36.300361 23468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:36.300585 23463 server_base.cc:1034] running on GCE node
W20260928 02:45:36.300379 23471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:36.301185 23463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:36.302358 23463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:36.303524 23463 hybrid_clock.cc:648] HybridClock initialized: now 1790563536303496 us; error 39 us; skew 500 ppm
Sep 28 02:45:36 dist-test-slave-rqvb krb5kdc[23450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:36.306345 23463 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:36.307242 23463 webserver.cc:460] Webserver started at http://127.0.0.1:38233/ using document root <none> and password file <none>
I20260928 02:45:36.307768 23463 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:36.307835 23463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:36.308030 23463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:36.310297 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "047108522f8a4262947a683d66518956"
format_stamp: "Formatted at 2026-09-28 02:45:36 on dist-test-slave-rqvb"
server_key: "a2c2ce2f4dac8fbaf7c4fca7731305d5"
server_key_iv: "9d7102eb9925eb291cf1f735a9d88e30"
server_key_version: "encryptionkey@0"
I20260928 02:45:36.310845 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "047108522f8a4262947a683d66518956"
format_stamp: "Formatted at 2026-09-28 02:45:36 on dist-test-slave-rqvb"
server_key: "a2c2ce2f4dac8fbaf7c4fca7731305d5"
server_key_iv: "9d7102eb9925eb291cf1f735a9d88e30"
server_key_version: "encryptionkey@0"
I20260928 02:45:36.314452 23463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:45:36.316834 23478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:36.318116 23463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:36.318225 23463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "047108522f8a4262947a683d66518956"
format_stamp: "Formatted at 2026-09-28 02:45:36 on dist-test-slave-rqvb"
server_key: "a2c2ce2f4dac8fbaf7c4fca7731305d5"
server_key_iv: "9d7102eb9925eb291cf1f735a9d88e30"
server_key_version: "encryptionkey@0"
I20260928 02:45:36.318320 23463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:36.328768 23463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:36.331725 23463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:36.331897 23463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:36.338586 23463 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37193
I20260928 02:45:36.338627 23530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37193 every 8 connection(s)
I20260928 02:45:36.339501 23463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:36.342252 23531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:36.346948 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23463
I20260928 02:45:36.347086 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:36.347332 2540 external_mini_cluster.cc:1428] Setting key 88e8e4056786a590ddeed68d59392fff
I20260928 02:45:36.347772 23531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956: Bootstrap starting.
I20260928 02:45:36.350257 23531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:36.351014 23531 log.cc:824] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:36.352828 23531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956: No bootstrap required, opened a new log
Sep 28 02:45:36 dist-test-slave-rqvb krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:45:36.356585 23531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047108522f8a4262947a683d66518956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260928 02:45:36.356880 23531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:36.356938 23531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047108522f8a4262947a683d66518956, State: Initialized, Role: FOLLOWER
I20260928 02:45:36.357437 23531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047108522f8a4262947a683d66518956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260928 02:45:36.357606 23531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:36.357710 23531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:36.357817 23531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:36.358719 23531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047108522f8a4262947a683d66518956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260928 02:45:36.359040 23531 leader_election.cc:302] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 047108522f8a4262947a683d66518956; no voters:
I20260928 02:45:36.359360 23531 leader_election.cc:288] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:36.359443 23536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:36.359881 23536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [term 1 LEADER]: Becoming Leader. State: Replica: 047108522f8a4262947a683d66518956, State: Running, Role: LEADER
I20260928 02:45:36.360230 23536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047108522f8a4262947a683d66518956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260928 02:45:36.360699 23531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:36.361922 23537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "047108522f8a4262947a683d66518956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047108522f8a4262947a683d66518956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } } }
I20260928 02:45:36.362061 23537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:36.362078 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:36.348791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:34555 (local address 127.0.0.1:37193)
0928 02:45:36.349245 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:36.349260 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:36.349295 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:45:36.350004 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:36.350026 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:36.350384 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:36.350739 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:36.351563 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:36.352535 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:36.353401 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:36.353874 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:36.354051 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:36.356687 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:36.356694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:36.356710 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:36.356742 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:36.358610 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:36.359147 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:36.359153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:36.359159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:36.359299 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:36.359696 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:36.359700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:36.359702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:36.360223 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:36.360434 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:36.360757 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:45:36.361120 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":143,"threads_started":1}
I20260928 02:45:36.362591 23538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 047108522f8a4262947a683d66518956. Latest consensus state: current_term: 1 leader_uuid: "047108522f8a4262947a683d66518956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047108522f8a4262947a683d66518956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } } }
I20260928 02:45:36.362743 23538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:36.365173 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 02:45:36.365707 23551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:36.365782 23551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:36.365839 23545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:36.366919 23545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:36.372305 23545 catalog_manager.cc:1347] Generated new cluster ID: 17ce02c7d991407192fae7419e3d8fd2
I20260928 02:45:36.372364 23545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:36.375255 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:36.367060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:55859 (local address 127.0.0.1:37193)
0928 02:45:36.367161 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:36.367165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:36.367454 (+ 289us) server_negotiation.cc:407] Connection header received
0928 02:45:36.367563 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:36.367573 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:36.367616 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:36.367673 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:36.368287 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:36.368857 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:36.370058 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:36.370386 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:36.370490 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:36.372230 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:36.372233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:36.372236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:36.372259 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:36.373588 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:36.374053 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:36.374058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:36.374059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:36.374141 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:36.374444 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:36.374448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:36.374449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:36.374616 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:36.374741 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:36.375004 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:45:36.375096 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:45:36.394572 23545 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:36.395551 23545 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:36.401095 23545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 047108522f8a4262947a683d66518956: Generated new TSK 0
I20260928 02:45:36.401769 23545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:36.411670 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23463
W20260928 02:45:36.415892 23448 connection.cc:535] client connection to 127.0.0.1:37193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37193): Connection reset by peer (error 104)
2026-09-28T02:45:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:36 dist-test-slave-rqvb krb5kdc[23562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:36 dist-test-slave-rqvb krb5kdc[23562](info): set up 2 sockets
Sep 28 02:45:36 dist-test-slave-rqvb krb5kdc[23562](info): commencing operation
krb5kdc: starting...
W20260928 02:45:37.373863 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:37 dist-test-slave-rqvb krb5kdc[23562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:37.489575 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:37.597148 23575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:37.597432 23575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:37.597491 23575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:37.601514 23575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:37.601586 23575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:37.601614 23575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:37.601635 23575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:37.601665 23575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:37.607097 23575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:37.608172 23575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:37.609196 23575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:37.617422 23581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:37.617555 23575 server_base.cc:1034] running on GCE node
W20260928 02:45:37.617416 23580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:37.617422 23583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:37.618418 23575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:37.619547 23575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:37.620741 23575 hybrid_clock.cc:648] HybridClock initialized: now 1790563537620723 us; error 44 us; skew 500 ppm
Sep 28 02:45:37 dist-test-slave-rqvb krb5kdc[23562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:37.627451 23575 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:37.628361 23575 webserver.cc:460] Webserver started at http://127.0.0.1:42997/ using document root <none> and password file <none>
I20260928 02:45:37.628899 23575 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:37.628964 23575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:37.629171 23575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:37.631292 23575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "6e46159a9eb04645b2c51256e6eda0c7"
format_stamp: "Formatted at 2026-09-28 02:45:37 on dist-test-slave-rqvb"
server_key: "10cf40d8ebad40c763c1c167dbe2daf1"
server_key_iv: "373ae57ba0eac99af2a7075c14001d47"
server_key_version: "encryptionkey@0"
I20260928 02:45:37.631837 23575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "6e46159a9eb04645b2c51256e6eda0c7"
format_stamp: "Formatted at 2026-09-28 02:45:37 on dist-test-slave-rqvb"
server_key: "10cf40d8ebad40c763c1c167dbe2daf1"
server_key_iv: "373ae57ba0eac99af2a7075c14001d47"
server_key_version: "encryptionkey@0"
I20260928 02:45:37.635600 23575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:45:37.638255 23590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:37.639546 23575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:37.639662 23575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "6e46159a9eb04645b2c51256e6eda0c7"
format_stamp: "Formatted at 2026-09-28 02:45:37 on dist-test-slave-rqvb"
server_key: "10cf40d8ebad40c763c1c167dbe2daf1"
server_key_iv: "373ae57ba0eac99af2a7075c14001d47"
server_key_version: "encryptionkey@0"
I20260928 02:45:37.639761 23575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:37.654479 23575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:37.657733 23575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:37.657925 23575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:37.664574 23575 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41099
I20260928 02:45:37.664685 23642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41099 every 8 connection(s)
I20260928 02:45:37.665542 23575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:37.668414 23643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:37.674060 23643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7: Bootstrap starting.
I20260928 02:45:37.675766 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23575
I20260928 02:45:37.675909 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:37.676170 2540 external_mini_cluster.cc:1428] Setting key 3ae56af2c1876aed49ebeb4df1c8f0db
I20260928 02:45:37.676718 23643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:37.677584 23643 log.cc:824] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:37.679920 23643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7: No bootstrap required, opened a new log
I20260928 02:45:37.684264 23643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e46159a9eb04645b2c51256e6eda0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41099 } }
I20260928 02:45:37.684566 23643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:37.684633 23643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e46159a9eb04645b2c51256e6eda0c7, State: Initialized, Role: FOLLOWER
I20260928 02:45:37.685160 23643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e46159a9eb04645b2c51256e6eda0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41099 } }
I20260928 02:45:37.685335 23643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:37.685405 23643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:37.685503 23643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:37.686293 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:37.677331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:57207 (local address 127.0.0.1:41099)
0928 02:45:37.677814 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:37.677836 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:37.677882 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:45:37.678728 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:37.678752 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:37.679184 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:37.679572 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:37.680001 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:37.681486 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:37.682434 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:37.683210 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:37.683389 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:37.684164 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:37.684171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:37.684187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:37.684721 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:37.684799 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:37.684831 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:37.684974 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:37.685143 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:45:37.685287 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":126,"threads_started":1}
I20260928 02:45:37.686728 23643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e46159a9eb04645b2c51256e6eda0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41099 } }
I20260928 02:45:37.687074 23643 leader_election.cc:302] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e46159a9eb04645b2c51256e6eda0c7; no voters:
I20260928 02:45:37.687444 23643 leader_election.cc:288] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:37.687510 23648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:37.687724 23648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [term 1 LEADER]: Becoming Leader. State: Replica: 6e46159a9eb04645b2c51256e6eda0c7, State: Running, Role: LEADER
I20260928 02:45:37.688072 23648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e46159a9eb04645b2c51256e6eda0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41099 } }
I20260928 02:45:37.688719 23643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:37.689652 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:37.689735 23649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e46159a9eb04645b2c51256e6eda0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e46159a9eb04645b2c51256e6eda0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41099 } } }
I20260928 02:45:37.689841 23649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:37.690316 23657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:37.690946 23650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e46159a9eb04645b2c51256e6eda0c7. Latest consensus state: current_term: 1 leader_uuid: "6e46159a9eb04645b2c51256e6eda0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e46159a9eb04645b2c51256e6eda0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41099 } } }
I20260928 02:45:37.691056 23650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:37.693622 23657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:37.697214 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:37.692162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:34181 (local address 127.0.0.1:41099)
0928 02:45:37.692266 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:37.692270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:37.692609 (+ 339us) server_negotiation.cc:407] Connection header received
0928 02:45:37.692792 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:37.692795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:37.692842 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:37.692888 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:37.693278 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:37.694056 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:37.695198 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:37.695526 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:37.695601 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:37.696740 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:37.696745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:37.696747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:37.696762 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:37.696793 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:37.696808 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:37.696911 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:37.697040 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:45:37.697078 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:45:37.699184 23657 catalog_manager.cc:1347] Generated new cluster ID: f8a72c51b6774c178eebb2197e15f693
I20260928 02:45:37.699241 23657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:37.709405 23657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:37.710421 23657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:37.718827 23657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e46159a9eb04645b2c51256e6eda0c7: Generated new TSK 0
I20260928 02:45:37.719523 23657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:37.724453 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23575
W20260928 02:45:37.728905 23560 connection.cc:535] client connection to 127.0.0.1:41099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41099): Connection reset by peer (error 104)
2026-09-28T02:45:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:37 dist-test-slave-rqvb krb5kdc[23674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:37 dist-test-slave-rqvb krb5kdc[23674](info): set up 2 sockets
Sep 28 02:45:37 dist-test-slave-rqvb krb5kdc[23674](info): commencing operation
krb5kdc: starting...
W20260928 02:45:38.681932 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:38 dist-test-slave-rqvb krb5kdc[23674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:38.795176 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:38.902941 23687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:38.903213 23687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:38.903272 23687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:38.907217 23687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:38.907286 23687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:38.907315 23687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:38.907336 23687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:38.907364 23687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:38.912582 23687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:38.913617 23687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:38.914628 23687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:38.923249 23692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:38.923290 23695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:38.923383 23687 server_base.cc:1034] running on GCE node
W20260928 02:45:38.923532 23693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:38.924050 23687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:38.925104 23687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:38.926265 23687 hybrid_clock.cc:648] HybridClock initialized: now 1790563538926235 us; error 40 us; skew 500 ppm
Sep 28 02:45:38 dist-test-slave-rqvb krb5kdc[23674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:38.928887 23687 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:38.929725 23687 webserver.cc:460] Webserver started at http://127.0.0.1:39571/ using document root <none> and password file <none>
I20260928 02:45:38.930280 23687 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:38.930346 23687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:38.930533 23687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:38.932534 23687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "34ac55788a7547d6b37f7073b72d09b0"
format_stamp: "Formatted at 2026-09-28 02:45:38 on dist-test-slave-rqvb"
server_key: "bd502b154d2f9c006dc8f4f5a61d249e"
server_key_iv: "9bb904407e8e9319a6afc79f613b7cdc"
server_key_version: "encryptionkey@0"
I20260928 02:45:38.933050 23687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "34ac55788a7547d6b37f7073b72d09b0"
format_stamp: "Formatted at 2026-09-28 02:45:38 on dist-test-slave-rqvb"
server_key: "bd502b154d2f9c006dc8f4f5a61d249e"
server_key_iv: "9bb904407e8e9319a6afc79f613b7cdc"
server_key_version: "encryptionkey@0"
I20260928 02:45:38.936918 23687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 02:45:38.939399 23702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:38.940618 23687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:38.940723 23687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "34ac55788a7547d6b37f7073b72d09b0"
format_stamp: "Formatted at 2026-09-28 02:45:38 on dist-test-slave-rqvb"
server_key: "bd502b154d2f9c006dc8f4f5a61d249e"
server_key_iv: "9bb904407e8e9319a6afc79f613b7cdc"
server_key_version: "encryptionkey@0"
I20260928 02:45:38.940846 23687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:38.945231 23687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:38.948024 23687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:38.948177 23687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:38.954963 23687 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38421
I20260928 02:45:38.954995 23754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38421 every 8 connection(s)
I20260928 02:45:38.955832 23687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:38.958532 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:38.961118 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23687
I20260928 02:45:38.961238 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:38.961488 2540 external_mini_cluster.cc:1428] Setting key 977a013f6705b62a47e2dedf8c370eb4
I20260928 02:45:38.964473 23755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0: Bootstrap starting.
I20260928 02:45:38.966794 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:38.962854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:54665 (local address 127.0.0.1:38421)
0928 02:45:38.963341 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:38.963365 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:38.963410 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:45:38.964052 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:38.964084 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:38.964374 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:38.964709 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:38.965519 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:38.965524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:38.965536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:38.965548 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:38.965613 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:38.965635 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:38.965754 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:38.965837 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:45:38.966008 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":128,"threads_started":1}
I20260928 02:45:38.966914 23755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:38.967625 23755 log.cc:824] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:38.969591 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:38.969794 23755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0: No bootstrap required, opened a new log
I20260928 02:45:38.973129 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:38.971475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:38975 (local address 127.0.0.1:38421)
0928 02:45:38.971607 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:38.971612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:38.971685 (+ 73us) server_negotiation.cc:407] Connection header received
0928 02:45:38.971800 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:38.971804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:38.971838 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:38.971926 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:38.972633 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:38.972637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:38.972640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:38.972654 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:38.972694 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:38.972709 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:38.972814 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:38.972916 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:45:38.972962 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:45:38.974279 23755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ac55788a7547d6b37f7073b72d09b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } }
I20260928 02:45:38.974522 23755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:38.974562 23755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ac55788a7547d6b37f7073b72d09b0, State: Initialized, Role: FOLLOWER
I20260928 02:45:38.974962 23755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ac55788a7547d6b37f7073b72d09b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } }
I20260928 02:45:38.975082 23755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:38.975121 23755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:38.975241 23755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:38.976178 23755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ac55788a7547d6b37f7073b72d09b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } }
I20260928 02:45:38.976501 23755 leader_election.cc:302] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ac55788a7547d6b37f7073b72d09b0; no voters:
I20260928 02:45:38.976819 23755 leader_election.cc:288] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:38.976908 23765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:38.977138 23765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [term 1 LEADER]: Becoming Leader. State: Replica: 34ac55788a7547d6b37f7073b72d09b0, State: Running, Role: LEADER
I20260928 02:45:38.977512 23765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ac55788a7547d6b37f7073b72d09b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } }
I20260928 02:45:38.978031 23755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:38.979044 23766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34ac55788a7547d6b37f7073b72d09b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ac55788a7547d6b37f7073b72d09b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } } }
I20260928 02:45:38.979157 23766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:38.979471 23767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ac55788a7547d6b37f7073b72d09b0. Latest consensus state: current_term: 1 leader_uuid: "34ac55788a7547d6b37f7073b72d09b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ac55788a7547d6b37f7073b72d09b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } } }
I20260928 02:45:38.979594 23767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:38.979523 23774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:38.982836 23774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:38.988344 23774 catalog_manager.cc:1347] Generated new cluster ID: 3f2b0f15344d4f86b70be02a926c2001
I20260928 02:45:38.988412 23774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:38.994365 23774 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:38.994705 23774 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:39.000882 23774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34ac55788a7547d6b37f7073b72d09b0: Generated new TSK 0
I20260928 02:45:39.001721 23774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:39.008154 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23687
2026-09-28T02:45:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:39 dist-test-slave-rqvb krb5kdc[23786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:39 dist-test-slave-rqvb krb5kdc[23786](info): set up 2 sockets
Sep 28 02:45:39 dist-test-slave-rqvb krb5kdc[23786](info): commencing operation
krb5kdc: starting...
W20260928 02:45:39.953903 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:40 dist-test-slave-rqvb krb5kdc[23786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:40.070791 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:40.180114 23799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:40.180404 23799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:40.180467 23799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:40.184470 23799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:40.184537 23799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:40.184562 23799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:40.184584 23799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:40.184612 23799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:40.189900 23799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:40.190979 23799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:40.192008 23799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:40.200109 23799 server_base.cc:1034] running on GCE node
W20260928 02:45:40.200033 23807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:40.200052 23805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:40.200064 23804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:40.200850 23799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:40.201930 23799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:40.203186 23799 hybrid_clock.cc:648] HybridClock initialized: now 1790563540203166 us; error 38 us; skew 500 ppm
Sep 28 02:45:40 dist-test-slave-rqvb krb5kdc[23786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:40.205888 23799 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:40.206791 23799 webserver.cc:460] Webserver started at http://127.0.0.1:41391/ using document root <none> and password file <none>
I20260928 02:45:40.207314 23799 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:40.207381 23799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:40.207571 23799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:40.209534 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "df4e6a00476841be82e728db2f8a8c89"
format_stamp: "Formatted at 2026-09-28 02:45:40 on dist-test-slave-rqvb"
server_key: "c9ea4f3c0144710b1f49ba18037e4cc4"
server_key_iv: "d049853cd577f348bd3c855021b383e8"
server_key_version: "encryptionkey@0"
I20260928 02:45:40.210094 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "df4e6a00476841be82e728db2f8a8c89"
format_stamp: "Formatted at 2026-09-28 02:45:40 on dist-test-slave-rqvb"
server_key: "c9ea4f3c0144710b1f49ba18037e4cc4"
server_key_iv: "d049853cd577f348bd3c855021b383e8"
server_key_version: "encryptionkey@0"
I20260928 02:45:40.213615 23799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:40.215981 23814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:40.217154 23799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:40.217262 23799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "df4e6a00476841be82e728db2f8a8c89"
format_stamp: "Formatted at 2026-09-28 02:45:40 on dist-test-slave-rqvb"
server_key: "c9ea4f3c0144710b1f49ba18037e4cc4"
server_key_iv: "d049853cd577f348bd3c855021b383e8"
server_key_version: "encryptionkey@0"
I20260928 02:45:40.217355 23799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:40.243893 23799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:40.247037 23799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:40.247226 23799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:40.254300 23799 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43795
I20260928 02:45:40.254319 23866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43795 every 8 connection(s)
I20260928 02:45:40.255266 23799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:40.256912 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23799
I20260928 02:45:40.257030 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:40.257304 2540 external_mini_cluster.cc:1428] Setting key e3c065162b6e5b2135639032295466ee
I20260928 02:45:40.258426 23867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:40.264587 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89: Bootstrap starting.
Sep 28 02:45:40 dist-test-slave-rqvb krb5kdc[23786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:45:40.267328 23867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:40.268144 23867 log.cc:824] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:40.270478 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89: No bootstrap required, opened a new log
I20260928 02:45:40.272974 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:40.258831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:45049 (local address 127.0.0.1:43795)
0928 02:45:40.259247 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:40.259266 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:40.259306 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:45:40.259952 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:40.259970 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:40.260320 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:40.260601 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:40.261540 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:40.262719 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:40.263913 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:40.264399 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:40.264516 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:40.267451 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:40.267458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:40.267472 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:40.267503 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:40.269472 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:40.270140 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:40.270147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:40.270153 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:40.270301 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:40.270686 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:40.270689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:40.270691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:40.271214 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:40.271417 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:40.271705 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:45:40.271954 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260928 02:45:40.274856 23867 raft_consensus.cc:348] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4e6a00476841be82e728db2f8a8c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43795 } }
I20260928 02:45:40.275137 23867 raft_consensus.cc:374] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:40.275195 23867 raft_consensus.cc:729] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4e6a00476841be82e728db2f8a8c89, State: Initialized, Role: FOLLOWER
I20260928 02:45:40.275647 23867 consensus_queue.cc:260] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4e6a00476841be82e728db2f8a8c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43795 } }
I20260928 02:45:40.275780 23867 raft_consensus.cc:388] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:40.275838 23867 raft_consensus.cc:482] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:40.275934 23867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:40.276080 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:40.277099 23867 raft_consensus.cc:504] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4e6a00476841be82e728db2f8a8c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43795 } }
I20260928 02:45:40.277498 23867 leader_election.cc:302] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df4e6a00476841be82e728db2f8a8c89; no voters:
I20260928 02:45:40.277853 23867 leader_election.cc:288] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:40.277946 23876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:40.278239 23876 raft_consensus.cc:686] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [term 1 LEADER]: Becoming Leader. State: Replica: df4e6a00476841be82e728db2f8a8c89, State: Running, Role: LEADER
I20260928 02:45:40.278601 23876 consensus_queue.cc:237] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4e6a00476841be82e728db2f8a8c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43795 } }
I20260928 02:45:40.279208 23867 sys_catalog.cc:564] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:40.279757 23876 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df4e6a00476841be82e728db2f8a8c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4e6a00476841be82e728db2f8a8c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43795 } } }
I20260928 02:45:40.279929 23876 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:40.280156 23876 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df4e6a00476841be82e728db2f8a8c89. Latest consensus state: current_term: 1 leader_uuid: "df4e6a00476841be82e728db2f8a8c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4e6a00476841be82e728db2f8a8c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43795 } } }
I20260928 02:45:40.280289 23876 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:40.280576 23884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:40.284384 23884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:40.287201 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:40.278316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:54315 (local address 127.0.0.1:43795)
0928 02:45:40.278417 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:40.278421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:40.278661 (+ 240us) server_negotiation.cc:407] Connection header received
0928 02:45:40.278816 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:40.278819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:40.278870 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:40.278955 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:40.279744 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:40.280518 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:40.281590 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:40.281922 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:40.282058 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:40.283598 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:40.283602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:40.283605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:40.283631 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:40.285196 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:40.285744 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:40.285747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:40.285749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:40.285835 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:40.286218 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:40.286221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:40.286222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:40.286438 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:40.286569 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:40.286843 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:45:40.286934 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:45:40.290169 23884 catalog_manager.cc:1347] Generated new cluster ID: 68965593b76e4fd58d9b6abcd0b92f81
I20260928 02:45:40.290237 23884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:40.300141 23884 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:40.301257 23884 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:40.309616 23884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df4e6a00476841be82e728db2f8a8c89: Generated new TSK 0
I20260928 02:45:40.310349 23884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:40.317435 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23799
W20260928 02:45:40.321939 23784 connection.cc:535] client connection to 127.0.0.1:43795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43795): Connection reset by peer (error 104)
2026-09-28T02:45:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:40 dist-test-slave-rqvb krb5kdc[23898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:40 dist-test-slave-rqvb krb5kdc[23898](info): set up 2 sockets
Sep 28 02:45:40 dist-test-slave-rqvb krb5kdc[23898](info): commencing operation
krb5kdc: starting...
W20260928 02:45:41.274259 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:41 dist-test-slave-rqvb krb5kdc[23898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:41.393167 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:41.499490 23911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:41.499751 23911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:41.499796 23911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:41.503831 23911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:41.503892 23911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:41.503911 23911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:41.503925 23911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:41.503938 23911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:41.509269 23911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:41.510389 23911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:41.511411 23911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:41.519965 23911 server_base.cc:1034] running on GCE node
W20260928 02:45:41.519919 23917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:41.519919 23919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:41.519919 23916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:41.520669 23911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:41.521836 23911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:41.523012 23911 hybrid_clock.cc:648] HybridClock initialized: now 1790563541522981 us; error 50 us; skew 500 ppm
Sep 28 02:45:41 dist-test-slave-rqvb krb5kdc[23898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:41.525718 23911 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:41.526613 23911 webserver.cc:460] Webserver started at http://127.0.0.1:36673/ using document root <none> and password file <none>
I20260928 02:45:41.527132 23911 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:41.527197 23911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:41.527396 23911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:41.529392 23911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "6146e4bac04a4cedb9a85b7a91623a75"
format_stamp: "Formatted at 2026-09-28 02:45:41 on dist-test-slave-rqvb"
server_key: "1284bf42de7cc839a298af4759e38f03"
server_key_iv: "e1241cdd4de576536f8320b807f5543b"
server_key_version: "encryptionkey@0"
I20260928 02:45:41.529932 23911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "6146e4bac04a4cedb9a85b7a91623a75"
format_stamp: "Formatted at 2026-09-28 02:45:41 on dist-test-slave-rqvb"
server_key: "1284bf42de7cc839a298af4759e38f03"
server_key_iv: "e1241cdd4de576536f8320b807f5543b"
server_key_version: "encryptionkey@0"
I20260928 02:45:41.533530 23911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:41.536031 23926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:41.537266 23911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:41.537375 23911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "6146e4bac04a4cedb9a85b7a91623a75"
format_stamp: "Formatted at 2026-09-28 02:45:41 on dist-test-slave-rqvb"
server_key: "1284bf42de7cc839a298af4759e38f03"
server_key_iv: "e1241cdd4de576536f8320b807f5543b"
server_key_version: "encryptionkey@0"
I20260928 02:45:41.537500 23911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:41.565744 23911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:41.568794 23911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:41.568969 23911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:41.575743 23911 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35085
I20260928 02:45:41.575814 23978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35085 every 8 connection(s)
I20260928 02:45:41.576691 23911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:41.579545 23979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:41.579701 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 23911
I20260928 02:45:41.579833 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:41.580128 2540 external_mini_cluster.cc:1428] Setting key 38ae9568f456e21388b2856d73c9a529
I20260928 02:45:41.585929 23979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75: Bootstrap starting.
I20260928 02:45:41.588183 23979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:41.588967 23979 log.cc:824] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:41.589260 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:41.581658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:43997 (local address 127.0.0.1:35085)
0928 02:45:41.582103 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:41.582118 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:41.582154 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:45:41.582846 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:41.582863 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:41.583210 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:41.583564 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:41.583944 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:41.584902 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:41.585792 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:41.586334 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:41.586487 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:41.587584 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:41.587589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:41.587602 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:41.587998 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:41.588053 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:41.588075 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:41.588188 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:41.588366 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:45:41.588508 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260928 02:45:41.590974 23979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75: No bootstrap required, opened a new log
I20260928 02:45:41.592200 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:41.594763 23979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6146e4bac04a4cedb9a85b7a91623a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35085 } }
I20260928 02:45:41.595078 23979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:41.595139 23979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6146e4bac04a4cedb9a85b7a91623a75, State: Initialized, Role: FOLLOWER
I20260928 02:45:41.595602 23979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6146e4bac04a4cedb9a85b7a91623a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35085 } }
I20260928 02:45:41.595753 23979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:41.595819 23979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:41.595924 23979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:41.597098 23979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6146e4bac04a4cedb9a85b7a91623a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35085 } }
I20260928 02:45:41.597472 23979 leader_election.cc:302] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6146e4bac04a4cedb9a85b7a91623a75; no voters:
I20260928 02:45:41.597810 23979 leader_election.cc:288] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:41.597891 23989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:41.598105 23989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [term 1 LEADER]: Becoming Leader. State: Replica: 6146e4bac04a4cedb9a85b7a91623a75, State: Running, Role: LEADER
I20260928 02:45:41.598255 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:41.594104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:37509 (local address 127.0.0.1:35085)
0928 02:45:41.594224 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:41.594228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:41.594362 (+ 134us) server_negotiation.cc:407] Connection header received
0928 02:45:41.594455 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:41.594458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:41.594505 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:41.594579 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:41.594792 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:41.595542 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:41.596459 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:41.596815 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:41.596920 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:41.597726 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:41.597730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:41.597733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:41.597746 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:41.597772 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:41.597783 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:41.597849 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:41.598079 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:45:41.598117 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:45:41.598472 23989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6146e4bac04a4cedb9a85b7a91623a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35085 } }
I20260928 02:45:41.599198 23979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:41.599685 23989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6146e4bac04a4cedb9a85b7a91623a75. Latest consensus state: current_term: 1 leader_uuid: "6146e4bac04a4cedb9a85b7a91623a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6146e4bac04a4cedb9a85b7a91623a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35085 } } }
I20260928 02:45:41.599798 23989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:41.600028 23989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6146e4bac04a4cedb9a85b7a91623a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6146e4bac04a4cedb9a85b7a91623a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35085 } } }
I20260928 02:45:41.600170 23989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:41.600221 23995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:41.603395 23995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:41.609828 23995 catalog_manager.cc:1347] Generated new cluster ID: 5cfb72acf5084dfc8073dd605a68d7b5
I20260928 02:45:41.609926 23995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:41.619869 23995 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:41.620833 23995 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:41.628249 23995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6146e4bac04a4cedb9a85b7a91623a75: Generated new TSK 0
I20260928 02:45:41.628929 23995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:41.641405 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 23911
W20260928 02:45:41.645773 23896 connection.cc:535] client connection to 127.0.0.1:35085 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35085): Connection reset by peer (error 104)
2026-09-28T02:45:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:41 dist-test-slave-rqvb krb5kdc[24010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:41 dist-test-slave-rqvb krb5kdc[24010](info): set up 2 sockets
Sep 28 02:45:41 dist-test-slave-rqvb krb5kdc[24010](info): commencing operation
krb5kdc: starting...
W20260928 02:45:42.597895 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:42 dist-test-slave-rqvb krb5kdc[24010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:45:42.713292 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37439
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:42.821425 24023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:42.821704 24023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:42.821762 24023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:42.826109 24023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:42.826201 24023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:42.826231 24023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:42.826253 24023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:42.826274 24023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:42.831750 24023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:42.832854 24023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:42.833913 24023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:42.842893 24028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:42.842926 24029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:42.842921 24031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:42.843014 24023 server_base.cc:1034] running on GCE node
I20260928 02:45:42.843863 24023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:42.845001 24023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:42.846158 24023 hybrid_clock.cc:648] HybridClock initialized: now 1790563542846139 us; error 35 us; skew 500 ppm
Sep 28 02:45:42 dist-test-slave-rqvb krb5kdc[24010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:42.854661 24023 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:45:42.855739 24023 webserver.cc:460] Webserver started at http://127.0.0.1:37179/ using document root <none> and password file <none>
I20260928 02:45:42.856334 24023 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:42.856400 24023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:42.856608 24023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:42.858875 24023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "12caa363a20246068d949111a3463bc5"
format_stamp: "Formatted at 2026-09-28 02:45:42 on dist-test-slave-rqvb"
server_key: "eae1257fa2fb85db0c9f5cdf3480ff72"
server_key_iv: "8acf743077b291a9c528b9368c6cab18"
server_key_version: "encryptionkey@0"
I20260928 02:45:42.859439 24023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "12caa363a20246068d949111a3463bc5"
format_stamp: "Formatted at 2026-09-28 02:45:42 on dist-test-slave-rqvb"
server_key: "eae1257fa2fb85db0c9f5cdf3480ff72"
server_key_iv: "8acf743077b291a9c528b9368c6cab18"
server_key_version: "encryptionkey@0"
I20260928 02:45:42.863184 24023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:45:42.865715 24038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:42.867152 24023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:45:42.867281 24023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "12caa363a20246068d949111a3463bc5"
format_stamp: "Formatted at 2026-09-28 02:45:42 on dist-test-slave-rqvb"
server_key: "eae1257fa2fb85db0c9f5cdf3480ff72"
server_key_iv: "8acf743077b291a9c528b9368c6cab18"
server_key_version: "encryptionkey@0"
I20260928 02:45:42.867383 24023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:42.871997 24023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:42.874971 24023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:42.875136 24023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:42.881845 24023 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37439
I20260928 02:45:42.881909 24090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37439 every 8 connection(s)
I20260928 02:45:42.882961 24023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:42.886230 24091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:42.889449 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24023
I20260928 02:45:42.889577 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:42.889839 2540 external_mini_cluster.cc:1428] Setting key c0cb0f5588d1aff126b576f51eaad558
I20260928 02:45:42.892624 24091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5: Bootstrap starting.
I20260928 02:45:42.895098 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:42.891338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:32785 (local address 127.0.0.1:37439)
0928 02:45:42.891757 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:42.891773 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:42.891808 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:45:42.892417 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:42.892433 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:42.892733 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:42.893045 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:42.893851 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:42.893856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:42.893867 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:42.893878 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:42.893929 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:42.893947 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:42.894096 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:42.894189 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:45:42.894340 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":129,"threads_started":1}
I20260928 02:45:42.895275 24091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:42.896126 24091 log.cc:824] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:42.897900 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:42.898391 24091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5: No bootstrap required, opened a new log
I20260928 02:45:42.901314 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:42.899815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.239:40833 (local address 127.0.0.1:37439)
0928 02:45:42.899932 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:42.899936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:42.900142 (+ 206us) server_negotiation.cc:407] Connection header received
0928 02:45:42.900247 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:42.900250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:42.900281 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:42.900340 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:42.900967 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:42.900971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:42.900973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:42.900984 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:42.901011 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:42.901022 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:42.901091 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:42.901163 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:45:42.901202 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:45:42.902079 24091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12caa363a20246068d949111a3463bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37439 } }
I20260928 02:45:42.902336 24091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:42.902388 24091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12caa363a20246068d949111a3463bc5, State: Initialized, Role: FOLLOWER
I20260928 02:45:42.902804 24091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12caa363a20246068d949111a3463bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37439 } }
I20260928 02:45:42.902925 24091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:42.902987 24091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:42.903072 24091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:42.903961 24091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12caa363a20246068d949111a3463bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37439 } }
I20260928 02:45:42.904284 24091 leader_election.cc:302] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12caa363a20246068d949111a3463bc5; no voters:
I20260928 02:45:42.904587 24091 leader_election.cc:288] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:42.904690 24101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:42.904922 24101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [term 1 LEADER]: Becoming Leader. State: Replica: 12caa363a20246068d949111a3463bc5, State: Running, Role: LEADER
I20260928 02:45:42.905206 24101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12caa363a20246068d949111a3463bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37439 } }
I20260928 02:45:42.905719 24091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:42.907661 24103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12caa363a20246068d949111a3463bc5. Latest consensus state: current_term: 1 leader_uuid: "12caa363a20246068d949111a3463bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12caa363a20246068d949111a3463bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37439 } } }
I20260928 02:45:42.907790 24103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:42.908069 24102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12caa363a20246068d949111a3463bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12caa363a20246068d949111a3463bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37439 } } }
I20260928 02:45:42.908380 24102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5 [sys.catalog]: This master's current role is: LEADER
W20260928 02:45:42.910442 24115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:42.910519 24115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:42.910573 24117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:42.911229 24117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:42.916615 24117 catalog_manager.cc:1347] Generated new cluster ID: 45da3d89b67d44bfbfd61e746161bff7
I20260928 02:45:42.916684 24117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:42.929039 24117 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:42.929397 24117 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:42.936048 24117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12caa363a20246068d949111a3463bc5: Generated new TSK 0
I20260928 02:45:42.936882 24117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:42.944478 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24023
2026-09-28T02:45:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:42 dist-test-slave-rqvb krb5kdc[24122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:42 dist-test-slave-rqvb krb5kdc[24122](info): set up 2 sockets
Sep 28 02:45:42 dist-test-slave-rqvb krb5kdc[24122](info): commencing operation
krb5kdc: starting...
W20260928 02:45:43.901822 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:43 dist-test-slave-rqvb krb5kdc[24122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:44.021373 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33429
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:44.130041 24135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:44.130342 24135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:44.130404 24135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:44.134505 24135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:44.134580 24135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:44.134605 24135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:44.134627 24135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:44.134656 24135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:44.140076 24135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:33429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:44.141185 24135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:44.142287 24135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:44.150770 24143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:44.150780 24141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:44.150827 24140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:44.150981 24135 server_base.cc:1034] running on GCE node
I20260928 02:45:44.151609 24135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:44.152771 24135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:44.153990 24135 hybrid_clock.cc:648] HybridClock initialized: now 1790563544153935 us; error 43 us; skew 500 ppm
Sep 28 02:45:44 dist-test-slave-rqvb krb5kdc[24122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563544, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:44.157049 24135 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:44.158155 24135 webserver.cc:460] Webserver started at http://127.2.123.62:34851/ using document root <none> and password file <none>
I20260928 02:45:44.158736 24135 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:44.158811 24135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:44.159017 24135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:44.161089 24135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "52792750be95470b8659f842201646f1"
format_stamp: "Formatted at 2026-09-28 02:45:44 on dist-test-slave-rqvb"
server_key: "be4477f4a09c009641d6260587898a7e"
server_key_iv: "b24d3db4849b1fb182e595d122c3ecde"
server_key_version: "encryptionkey@0"
I20260928 02:45:44.161633 24135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "52792750be95470b8659f842201646f1"
format_stamp: "Formatted at 2026-09-28 02:45:44 on dist-test-slave-rqvb"
server_key: "be4477f4a09c009641d6260587898a7e"
server_key_iv: "b24d3db4849b1fb182e595d122c3ecde"
server_key_version: "encryptionkey@0"
I20260928 02:45:44.165349 24135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:45:44.167873 24150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:44.169204 24135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:44.169322 24135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "52792750be95470b8659f842201646f1"
format_stamp: "Formatted at 2026-09-28 02:45:44 on dist-test-slave-rqvb"
server_key: "be4477f4a09c009641d6260587898a7e"
server_key_iv: "b24d3db4849b1fb182e595d122c3ecde"
server_key_version: "encryptionkey@0"
I20260928 02:45:44.169422 24135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:44.184720 24135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:44.187914 24135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:44.188097 24135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:44.194913 24135 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:33429
I20260928 02:45:44.194936 24202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:33429 every 8 connection(s)
I20260928 02:45:44.195838 24135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:44.197661 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24135
I20260928 02:45:44.197836 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:44.198217 2540 external_mini_cluster.cc:1428] Setting key 946e5dde8ab62abc6bfc0c2fada3a054
I20260928 02:45:44.198889 24203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:44.205907 24203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1: Bootstrap starting.
Sep 28 02:45:44 dist-test-slave-rqvb krb5kdc[24122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:44.208611 24203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:44.209483 24203 log.cc:824] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:44.211796 24203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1: No bootstrap required, opened a new log
I20260928 02:45:44.213832 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:44.199798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34409 (local address 127.2.123.62:33429)
0928 02:45:44.200224 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:44.200241 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:44.200277 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:45:44.200970 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:44.200988 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:44.201332 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:44.201636 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:44.202665 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:44.203744 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:44.204932 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:44.205464 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:44.205602 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:44.208406 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:44.208413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:44.208430 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:44.208462 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:44.210308 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:44.210855 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:44.210860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:44.210866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:44.211018 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:44.211386 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:44.211389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:44.211391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:44.211903 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:44.212116 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:44.212404 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:45:44.212630 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
I20260928 02:45:44.215765 24203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52792750be95470b8659f842201646f1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33429 } }
I20260928 02:45:44.216065 24203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:44.216138 24203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52792750be95470b8659f842201646f1, State: Initialized, Role: FOLLOWER
I20260928 02:45:44.216603 24203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52792750be95470b8659f842201646f1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33429 } }
I20260928 02:45:44.216738 24203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:44.216809 24203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:44.216907 24203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:44.217015 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:44.218127 24203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52792750be95470b8659f842201646f1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33429 } }
I20260928 02:45:44.218534 24203 leader_election.cc:302] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52792750be95470b8659f842201646f1; no voters:
I20260928 02:45:44.218878 24203 leader_election.cc:288] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:44.218997 24212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:44.219237 24212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [term 1 LEADER]: Becoming Leader. State: Replica: 52792750be95470b8659f842201646f1, State: Running, Role: LEADER
I20260928 02:45:44.219602 24212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52792750be95470b8659f842201646f1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33429 } }
I20260928 02:45:44.220155 24203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:44.220798 24212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52792750be95470b8659f842201646f1. Latest consensus state: current_term: 1 leader_uuid: "52792750be95470b8659f842201646f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52792750be95470b8659f842201646f1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33429 } } }
I20260928 02:45:44.220963 24212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:44.221194 24214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52792750be95470b8659f842201646f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52792750be95470b8659f842201646f1" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 33429 } } }
I20260928 02:45:44.221354 24214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:44.221405 24222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:44.224746 24222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:44.230556 24222 catalog_manager.cc:1347] Generated new cluster ID: 5b81d57fad0c499e875a733997527c7c
I20260928 02:45:44.230686 24222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:44.231606 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:44.219265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41079 (local address 127.2.123.62:33429)
0928 02:45:44.219372 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:44.219376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:44.222319 (+ 2943us) server_negotiation.cc:407] Connection header received
0928 02:45:44.222566 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:44.222570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:44.222621 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:44.222708 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:44.223524 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:44.224263 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:44.225599 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:44.225938 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:44.226069 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:44.227707 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:44.227710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:44.227713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:44.227739 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:44.229328 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:44.229866 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:44.229869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:44.229871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:44.229984 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:44.230338 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:44.230342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:44.230344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:44.230566 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:44.230704 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:44.231354 (+ 650us) server_negotiation.cc:299] Negotiation successful
0928 02:45:44.231450 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:45:44.242928 24222 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:44.243922 24222 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:44.256868 24222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52792750be95470b8659f842201646f1: Generated new TSK 0
I20260928 02:45:44.257653 24222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:44.270211 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24135
W20260928 02:45:44.274744 24120 connection.cc:535] client connection to 127.2.123.62:33429 recv error: Network error: failed to read from TLS socket (remote: 127.2.123.62:33429): Connection reset by peer (error 104)
2026-09-28T02:45:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:44 dist-test-slave-rqvb krb5kdc[24234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:44 dist-test-slave-rqvb krb5kdc[24234](info): set up 2 sockets
Sep 28 02:45:44 dist-test-slave-rqvb krb5kdc[24234](info): commencing operation
krb5kdc: starting...
W20260928 02:45:45.234114 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:45 dist-test-slave-rqvb krb5kdc[24234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:45.352522 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:34329
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:34329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:45.458738 24247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:45.458993 24247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:45.459033 24247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:45.462957 24247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:45.463013 24247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:45.463032 24247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:45.463047 24247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:45.463060 24247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:45.468271 24247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:34329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:45.469249 24247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:45.470211 24247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:45.478193 24247 server_base.cc:1034] running on GCE node
W20260928 02:45:45.478294 24253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:45.478231 24255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:45.478231 24252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:45.479048 24247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:45.480126 24247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:45.481272 24247 hybrid_clock.cc:648] HybridClock initialized: now 1790563545481255 us; error 39 us; skew 500 ppm
Sep 28 02:45:45 dist-test-slave-rqvb krb5kdc[24234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:45.483981 24247 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:45.484841 24247 webserver.cc:460] Webserver started at http://127.2.123.62:39027/ using document root <none> and password file <none>
I20260928 02:45:45.485369 24247 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:45.485437 24247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:45.485620 24247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:45.487643 24247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "44c865c9e3c642368076a51a2cee35ad"
format_stamp: "Formatted at 2026-09-28 02:45:45 on dist-test-slave-rqvb"
server_key: "27abc697ddf33786a8ea8d188661d501"
server_key_iv: "547a4b374b87a3afa07536632bd350e8"
server_key_version: "encryptionkey@0"
I20260928 02:45:45.488173 24247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "44c865c9e3c642368076a51a2cee35ad"
format_stamp: "Formatted at 2026-09-28 02:45:45 on dist-test-slave-rqvb"
server_key: "27abc697ddf33786a8ea8d188661d501"
server_key_iv: "547a4b374b87a3afa07536632bd350e8"
server_key_version: "encryptionkey@0"
I20260928 02:45:45.491748 24247 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 02:45:45.494058 24262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:45.495232 24247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:45.495339 24247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "44c865c9e3c642368076a51a2cee35ad"
format_stamp: "Formatted at 2026-09-28 02:45:45 on dist-test-slave-rqvb"
server_key: "27abc697ddf33786a8ea8d188661d501"
server_key_iv: "547a4b374b87a3afa07536632bd350e8"
server_key_version: "encryptionkey@0"
I20260928 02:45:45.495430 24247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:45.512645 24247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:45.515726 24247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:45.515909 24247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:45.522624 24247 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:34329
I20260928 02:45:45.522679 24314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:34329 every 8 connection(s)
I20260928 02:45:45.523542 24247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:45.526320 24315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:45.528481 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24247
I20260928 02:45:45.528618 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:45.528905 2540 external_mini_cluster.cc:1428] Setting key 0d81ecbdf7d91dac82c0a732ac4bff2b
I20260928 02:45:45.532289 24315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad: Bootstrap starting.
I20260928 02:45:45.534421 24315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:45.535130 24315 log.cc:824] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:45.536899 24315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad: No bootstrap required, opened a new log
I20260928 02:45:45.537771 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:45.530323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41971 (local address 127.2.123.62:34329)
0928 02:45:45.530740 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:45.530754 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:45.530788 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:45:45.531489 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:45.531506 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:45.531834 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:45.532137 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:45.532535 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:45.533479 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:45.534384 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:45.534877 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:45.535024 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:45.536033 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:45.536038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:45.536051 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:45.536462 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:45.536516 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:45.536538 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:45.536654 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:45.536935 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:45:45.537080 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":137,"threads_started":1}
I20260928 02:45:45.540695 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:45.540751 24315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c865c9e3c642368076a51a2cee35ad" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 34329 } }
I20260928 02:45:45.540983 24315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:45.541036 24315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c865c9e3c642368076a51a2cee35ad, State: Initialized, Role: FOLLOWER
I20260928 02:45:45.541424 24315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c865c9e3c642368076a51a2cee35ad" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 34329 } }
I20260928 02:45:45.541538 24315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:45.541592 24315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:45.541682 24315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:45.542611 24315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c865c9e3c642368076a51a2cee35ad" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 34329 } }
I20260928 02:45:45.542938 24315 leader_election.cc:302] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c865c9e3c642368076a51a2cee35ad; no voters:
I20260928 02:45:45.543213 24315 leader_election.cc:288] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:45.543324 24325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:45.543545 24325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [term 1 LEADER]: Becoming Leader. State: Replica: 44c865c9e3c642368076a51a2cee35ad, State: Running, Role: LEADER
I20260928 02:45:45.543906 24325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c865c9e3c642368076a51a2cee35ad" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 34329 } }
I20260928 02:45:45.544440 24315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:45.545032 24326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c865c9e3c642368076a51a2cee35ad. Latest consensus state: current_term: 1 leader_uuid: "44c865c9e3c642368076a51a2cee35ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c865c9e3c642368076a51a2cee35ad" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 34329 } } }
I20260928 02:45:45.545146 24326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:45.545452 24327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c865c9e3c642368076a51a2cee35ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c865c9e3c642368076a51a2cee35ad" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 34329 } } }
I20260928 02:45:45.545542 24327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:45.545569 24331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:45.547509 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:45.542704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60461 (local address 127.2.123.62:34329)
0928 02:45:45.542846 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:45.542850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:45.542990 (+ 140us) server_negotiation.cc:407] Connection header received
0928 02:45:45.543117 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:45.543120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:45.543168 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:45.543241 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:45.543520 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:45.544286 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:45.545631 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:45.545995 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:45.546091 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:45.547077 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:45.547081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:45.547083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:45.547095 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:45.547121 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:45.547133 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:45.547230 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:45.547346 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 02:45:45.547377 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":64}
I20260928 02:45:45.548929 24331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:45.554507 24331 catalog_manager.cc:1347] Generated new cluster ID: b1b838d38f244c4fb67a5e68536ae463
I20260928 02:45:45.554577 24331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:45.563953 24331 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:45.565128 24331 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:45.571142 24331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c865c9e3c642368076a51a2cee35ad: Generated new TSK 0
I20260928 02:45:45.571977 24331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:45.581218 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24247
W20260928 02:45:45.585664 24232 connection.cc:535] client connection to 127.2.123.62:34329 recv error: Network error: failed to read from TLS socket (remote: 127.2.123.62:34329): Connection reset by peer (error 104)
2026-09-28T02:45:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:45 dist-test-slave-rqvb krb5kdc[24346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:45 dist-test-slave-rqvb krb5kdc[24346](info): set up 2 sockets
Sep 28 02:45:45 dist-test-slave-rqvb krb5kdc[24346](info): commencing operation
krb5kdc: starting...
W20260928 02:45:46.529843 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:46 dist-test-slave-rqvb krb5kdc[24346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:46.644912 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45557
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:45557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:46.751799 24359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:46.752063 24359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:46.752108 24359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:46.756080 24359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:46.756135 24359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:46.756152 24359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:46.756168 24359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:46.756192 24359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:46.761626 24359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:45557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:46.762674 24359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:46.763664 24359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:46.772051 24367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:46.772184 24364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:46.772261 24365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:46.772325 24359 server_base.cc:1034] running on GCE node
I20260928 02:45:46.772910 24359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:46.774121 24359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:46.775311 24359 hybrid_clock.cc:648] HybridClock initialized: now 1790563546775280 us; error 46 us; skew 500 ppm
Sep 28 02:45:46 dist-test-slave-rqvb krb5kdc[24346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:46.778236 24359 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:46.779233 24359 webserver.cc:460] Webserver started at http://127.2.123.62:36145/ using document root <none> and password file <none>
I20260928 02:45:46.779764 24359 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:46.779831 24359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:46.780041 24359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:46.782083 24359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "a3afd3d482184a64a2a44a83c41bc724"
format_stamp: "Formatted at 2026-09-28 02:45:46 on dist-test-slave-rqvb"
server_key: "60c40638f68061f2792f81c7c01d6311"
server_key_iv: "77147c36e5d8c5bcac6624178f101d23"
server_key_version: "encryptionkey@0"
I20260928 02:45:46.782631 24359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "a3afd3d482184a64a2a44a83c41bc724"
format_stamp: "Formatted at 2026-09-28 02:45:46 on dist-test-slave-rqvb"
server_key: "60c40638f68061f2792f81c7c01d6311"
server_key_iv: "77147c36e5d8c5bcac6624178f101d23"
server_key_version: "encryptionkey@0"
I20260928 02:45:46.786273 24359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:45:46.788679 24374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:46.789954 24359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:46.790114 24359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "a3afd3d482184a64a2a44a83c41bc724"
format_stamp: "Formatted at 2026-09-28 02:45:46 on dist-test-slave-rqvb"
server_key: "60c40638f68061f2792f81c7c01d6311"
server_key_iv: "77147c36e5d8c5bcac6624178f101d23"
server_key_version: "encryptionkey@0"
I20260928 02:45:46.790294 24359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:46.794934 24359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:46.797854 24359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:46.798039 24359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:46.804484 24359 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:45557
I20260928 02:45:46.804502 24426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:45557 every 8 connection(s)
I20260928 02:45:46.805361 24359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:46.808038 24427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:46.810626 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24359
I20260928 02:45:46.810755 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:46.811007 2540 external_mini_cluster.cc:1428] Setting key 4aee2c12dcaa4bd85305abedea37493b
I20260928 02:45:46.813642 24427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724: Bootstrap starting.
I20260928 02:45:46.815905 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:46.812257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47315 (local address 127.2.123.62:45557)
0928 02:45:46.812644 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:46.812658 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:46.812692 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:45:46.813292 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:46.813308 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:46.813582 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:46.813878 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:46.814619 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:46.814624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:46.814635 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:46.814644 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:46.814696 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:46.814768 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:46.814915 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:46.814984 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:45:46.815129 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260928 02:45:46.816087 24427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:46.816787 24427 log.cc:824] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:46.818626 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:46.818643 24427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724: No bootstrap required, opened a new log
I20260928 02:45:46.822454 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:46.820814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41755 (local address 127.2.123.62:45557)
0928 02:45:46.820928 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:46.820932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:46.821184 (+ 252us) server_negotiation.cc:407] Connection header received
0928 02:45:46.821274 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:46.821276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:46.821306 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:46.821372 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:46.822102 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:46.822105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:46.822107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:46.822117 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:46.822141 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:46.822151 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:46.822213 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:46.822310 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:45:46.822340 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:45:46.822904 24427 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3afd3d482184a64a2a44a83c41bc724" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45557 } }
I20260928 02:45:46.823124 24427 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:46.823184 24427 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3afd3d482184a64a2a44a83c41bc724, State: Initialized, Role: FOLLOWER
I20260928 02:45:46.823592 24427 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3afd3d482184a64a2a44a83c41bc724" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45557 } }
I20260928 02:45:46.823717 24427 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:46.823812 24427 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:46.823961 24427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:46.824869 24427 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3afd3d482184a64a2a44a83c41bc724" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45557 } }
I20260928 02:45:46.825186 24427 leader_election.cc:302] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3afd3d482184a64a2a44a83c41bc724; no voters:
I20260928 02:45:46.825464 24427 leader_election.cc:288] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:46.825605 24437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:46.825807 24437 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [term 1 LEADER]: Becoming Leader. State: Replica: a3afd3d482184a64a2a44a83c41bc724, State: Running, Role: LEADER
I20260928 02:45:46.826114 24437 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3afd3d482184a64a2a44a83c41bc724" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45557 } }
I20260928 02:45:46.826655 24427 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:46.827735 24438 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3afd3d482184a64a2a44a83c41bc724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3afd3d482184a64a2a44a83c41bc724" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45557 } } }
I20260928 02:45:46.827837 24438 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:46.828128 24439 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3afd3d482184a64a2a44a83c41bc724. Latest consensus state: current_term: 1 leader_uuid: "a3afd3d482184a64a2a44a83c41bc724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3afd3d482184a64a2a44a83c41bc724" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45557 } } }
I20260928 02:45:46.828217 24446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:46.828222 24439 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:46.831315 24446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:46.837121 24446 catalog_manager.cc:1347] Generated new cluster ID: e93999819146461b99d20b3226dabe37
I20260928 02:45:46.837194 24446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:46.850541 24446 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:46.850942 24446 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:46.864354 24446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3afd3d482184a64a2a44a83c41bc724: Generated new TSK 0
I20260928 02:45:46.865324 24446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:46.870667 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24359
2026-09-28T02:45:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:46 dist-test-slave-rqvb krb5kdc[24458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:46 dist-test-slave-rqvb krb5kdc[24458](info): set up 2 sockets
Sep 28 02:45:46 dist-test-slave-rqvb krb5kdc[24458](info): commencing operation
krb5kdc: starting...
W20260928 02:45:47.818270 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:47 dist-test-slave-rqvb krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:47.932224 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:37847
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:44741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:37847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:48.038719 24471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:48.038988 24471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:48.039032 24471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:48.043016 24471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:48.043078 24471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:48.043095 24471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:48.043110 24471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:48.043124 24471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:48.048444 24471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:44741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:37847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:37847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:48.049463 24471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:48.050473 24471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:48.058769 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:48.058804 24476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:48.058979 24471 server_base.cc:1034] running on GCE node
W20260928 02:45:48.058996 24479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:48.059638 24471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:48.060781 24471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:48.061990 24471 hybrid_clock.cc:648] HybridClock initialized: now 1790563548061945 us; error 32 us; skew 500 ppm
Sep 28 02:45:48 dist-test-slave-rqvb krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563548, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:48.070489 24471 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:48.071537 24471 webserver.cc:460] Webserver started at http://127.2.123.62:46753/ using document root <none> and password file <none>
I20260928 02:45:48.072103 24471 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:48.072171 24471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:48.072378 24471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:48.074519 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "f73d73dc6e2e45ef865952710ba23d84"
format_stamp: "Formatted at 2026-09-28 02:45:48 on dist-test-slave-rqvb"
server_key: "9323b14ab66bfd3bd73796ff26ee0a33"
server_key_iv: "75638dfbd2b1a95e70b2ebaccf295055"
server_key_version: "encryptionkey@0"
I20260928 02:45:48.075073 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "f73d73dc6e2e45ef865952710ba23d84"
format_stamp: "Formatted at 2026-09-28 02:45:48 on dist-test-slave-rqvb"
server_key: "9323b14ab66bfd3bd73796ff26ee0a33"
server_key_iv: "75638dfbd2b1a95e70b2ebaccf295055"
server_key_version: "encryptionkey@0"
I20260928 02:45:48.078926 24471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:45:48.081585 24486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:48.083046 24471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:48.083170 24471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "f73d73dc6e2e45ef865952710ba23d84"
format_stamp: "Formatted at 2026-09-28 02:45:48 on dist-test-slave-rqvb"
server_key: "9323b14ab66bfd3bd73796ff26ee0a33"
server_key_iv: "75638dfbd2b1a95e70b2ebaccf295055"
server_key_version: "encryptionkey@0"
I20260928 02:45:48.083287 24471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:48.101460 24471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:48.105157 24471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:48.105360 24471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:48.112479 24471 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:37847
I20260928 02:45:48.112504 24538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:37847 every 8 connection(s)
I20260928 02:45:48.113523 24471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:48.116456 24539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:48.118407 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24471
I20260928 02:45:48.118554 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:48.118871 2540 external_mini_cluster.cc:1428] Setting key b9099b609c41d711fd1dbcd50cc42019
I20260928 02:45:48.122567 24539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84: Bootstrap starting.
I20260928 02:45:48.124716 24539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:48.125427 24539 log.cc:824] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:48.127661 24539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84: No bootstrap required, opened a new log
Sep 28 02:45:48 dist-test-slave-rqvb krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:48.131019 24539 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d73dc6e2e45ef865952710ba23d84" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37847 } }
I20260928 02:45:48.131296 24539 raft_consensus.cc:374] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:48.131358 24539 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73d73dc6e2e45ef865952710ba23d84, State: Initialized, Role: FOLLOWER
I20260928 02:45:48.131795 24539 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d73dc6e2e45ef865952710ba23d84" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37847 } }
I20260928 02:45:48.131927 24539 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:48.131991 24539 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:48.132085 24539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:48.133071 24539 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d73dc6e2e45ef865952710ba23d84" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37847 } }
I20260928 02:45:48.133384 24539 leader_election.cc:302] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73d73dc6e2e45ef865952710ba23d84; no voters:
I20260928 02:45:48.133545 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:48.120286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33759 (local address 127.2.123.62:37847)
0928 02:45:48.120775 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:48.120799 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:48.120842 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:45:48.121644 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:48.121662 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:48.122041 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:48.122377 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:48.123198 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:48.124177 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:48.125028 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:48.125497 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:48.125680 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:48.128429 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:48.128435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:48.128448 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:48.128474 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:48.130580 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:48.131124 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:48.131128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:48.131134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:48.131248 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:48.131531 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:48.131535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:48.131537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:48.132005 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:48.132168 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:48.132440 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:45:48.132785 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":133,"threads_started":1}
I20260928 02:45:48.133678 24539 leader_election.cc:288] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:48.133800 24544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:48.134019 24544 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [term 1 LEADER]: Becoming Leader. State: Replica: f73d73dc6e2e45ef865952710ba23d84, State: Running, Role: LEADER
I20260928 02:45:48.134356 24544 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d73dc6e2e45ef865952710ba23d84" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37847 } }
I20260928 02:45:48.134933 24539 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:48.135551 24546 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73d73dc6e2e45ef865952710ba23d84. Latest consensus state: current_term: 1 leader_uuid: "f73d73dc6e2e45ef865952710ba23d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d73dc6e2e45ef865952710ba23d84" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37847 } } }
I20260928 02:45:48.135661 24546 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:48.136345 24545 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f73d73dc6e2e45ef865952710ba23d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73d73dc6e2e45ef865952710ba23d84" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37847 } } }
I20260928 02:45:48.136494 24545 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:48.136693 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:48.137075 24552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:48.140143 24552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:48.145725 24552 catalog_manager.cc:1347] Generated new cluster ID: 668928826d5f411481239e2a57fad1b6
I20260928 02:45:48.145784 24552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:48.147571 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:48.138818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43389 (local address 127.2.123.62:37847)
0928 02:45:48.138920 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:48.138924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:48.139166 (+ 242us) server_negotiation.cc:407] Connection header received
0928 02:45:48.139270 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:48.139273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:48.139316 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:48.139395 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:48.140135 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:48.140895 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:48.142081 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:48.142439 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:48.142540 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:48.144245 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:48.144249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:48.144252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:48.144280 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:48.145885 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:48.146382 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:48.146386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:48.146388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:48.146477 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:48.146741 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:48.146745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:48.146747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:48.146966 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:48.147094 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:48.147286 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:45:48.147412 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:45:48.152762 24552 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:48.153676 24552 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:48.157887 24552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f73d73dc6e2e45ef865952710ba23d84: Generated new TSK 0
I20260928 02:45:48.158515 24552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:48.165043 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24471
2026-09-28T02:45:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:48 dist-test-slave-rqvb krb5kdc[24570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:48 dist-test-slave-rqvb krb5kdc[24570](info): set up 2 sockets
Sep 28 02:45:48 dist-test-slave-rqvb krb5kdc[24570](info): commencing operation
krb5kdc: starting...
W20260928 02:45:49.117944 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:49 dist-test-slave-rqvb krb5kdc[24570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:49.237927 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:35253
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:49.347846 24583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:49.348134 24583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:49.348194 24583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:49.352231 24583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:49.352298 24583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:49.352324 24583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:49.352344 24583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:49.352365 24583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:49.357547 24583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:49.358577 24583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:49.359571 24583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:49.368108 24583 server_base.cc:1034] running on GCE node
W20260928 02:45:49.368070 24588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:49.368065 24589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:49.368070 24591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:49.368933 24583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:49.370092 24583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:49.371281 24583 hybrid_clock.cc:648] HybridClock initialized: now 1790563549371258 us; error 34 us; skew 500 ppm
Sep 28 02:45:49 dist-test-slave-rqvb krb5kdc[24570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563549, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:49.374133 24583 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:49.375043 24583 webserver.cc:460] Webserver started at http://127.2.123.62:44499/ using document root <none> and password file <none>
I20260928 02:45:49.375567 24583 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:49.375633 24583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:49.375839 24583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:49.377862 24583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "a1b7352d143b46d9807164ca01d02efb"
format_stamp: "Formatted at 2026-09-28 02:45:49 on dist-test-slave-rqvb"
server_key: "8fd7efa426704ef79324285674408915"
server_key_iv: "9b4d2759918022a2943cda2fac199065"
server_key_version: "encryptionkey@0"
I20260928 02:45:49.378453 24583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "a1b7352d143b46d9807164ca01d02efb"
format_stamp: "Formatted at 2026-09-28 02:45:49 on dist-test-slave-rqvb"
server_key: "8fd7efa426704ef79324285674408915"
server_key_iv: "9b4d2759918022a2943cda2fac199065"
server_key_version: "encryptionkey@0"
I20260928 02:45:49.382200 24583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:45:49.384575 24598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:49.385815 24583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:49.385946 24583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "a1b7352d143b46d9807164ca01d02efb"
format_stamp: "Formatted at 2026-09-28 02:45:49 on dist-test-slave-rqvb"
server_key: "8fd7efa426704ef79324285674408915"
server_key_iv: "9b4d2759918022a2943cda2fac199065"
server_key_version: "encryptionkey@0"
I20260928 02:45:49.386102 24583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:49.398695 24583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:49.401826 24583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:49.402041 24583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:49.408633 24583 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:35253
I20260928 02:45:49.408650 24650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:35253 every 8 connection(s)
I20260928 02:45:49.409569 24583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:49.412320 24651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:49.414383 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24583
I20260928 02:45:49.414516 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:49.414762 2540 external_mini_cluster.cc:1428] Setting key a5fdc58e0c5a64ddb90e027c5e6aa33f
I20260928 02:45:49.418092 24651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb: Bootstrap starting.
I20260928 02:45:49.420356 24651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:49.421056 24651 log.cc:824] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:49.422993 24651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb: No bootstrap required, opened a new log
I20260928 02:45:49.423359 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:49.416101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51157 (local address 127.2.123.62:35253)
0928 02:45:49.416535 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:49.416550 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:49.416585 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:45:49.417212 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:49.417229 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:49.417596 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:49.417903 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:49.418315 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:49.419270 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:49.420131 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:49.420689 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:49.420831 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:49.421646 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:49.421653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:49.421671 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:49.422102 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:49.422154 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:49.422177 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:49.422251 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:49.422420 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:45:49.422622 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":133,"threads_started":1}
I20260928 02:45:49.425900 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:49.426492 24651 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b7352d143b46d9807164ca01d02efb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35253 } }
I20260928 02:45:49.426769 24651 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:49.426833 24651 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b7352d143b46d9807164ca01d02efb, State: Initialized, Role: FOLLOWER
I20260928 02:45:49.427311 24651 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b7352d143b46d9807164ca01d02efb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35253 } }
I20260928 02:45:49.427472 24651 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:49.427538 24651 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:49.427634 24651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:49.428664 24651 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b7352d143b46d9807164ca01d02efb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35253 } }
I20260928 02:45:49.429004 24651 leader_election.cc:302] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b7352d143b46d9807164ca01d02efb; no voters:
I20260928 02:45:49.429308 24651 leader_election.cc:288] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:49.429389 24661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:49.429600 24661 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [term 1 LEADER]: Becoming Leader. State: Replica: a1b7352d143b46d9807164ca01d02efb, State: Running, Role: LEADER
I20260928 02:45:49.429931 24661 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b7352d143b46d9807164ca01d02efb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35253 } }
I20260928 02:45:49.430562 24651 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:49.431475 24662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b7352d143b46d9807164ca01d02efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b7352d143b46d9807164ca01d02efb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35253 } } }
I20260928 02:45:49.431861 24662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:49.432237 24663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b7352d143b46d9807164ca01d02efb. Latest consensus state: current_term: 1 leader_uuid: "a1b7352d143b46d9807164ca01d02efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b7352d143b46d9807164ca01d02efb" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 35253 } } }
I20260928 02:45:49.432490 24663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:49.432582 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:49.427996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60205 (local address 127.2.123.62:35253)
0928 02:45:49.428108 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:49.428112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:49.428269 (+ 157us) server_negotiation.cc:407] Connection header received
0928 02:45:49.428353 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:49.428356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:49.428407 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:49.428491 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:49.428824 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:49.429576 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:49.430640 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:49.430966 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:49.431064 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:49.431955 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:49.431959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:49.431962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:49.431973 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:49.432001 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:49.432015 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:49.432078 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:49.432377 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:45:49.432427 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 02:45:49.435288 24677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:49.435355 24677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:49.435426 24670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:49.436154 24670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:49.441221 24670 catalog_manager.cc:1347] Generated new cluster ID: 187286f68f43406db14ad5a725b633ac
I20260928 02:45:49.441298 24670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:49.453094 24670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:49.454049 24670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:49.458967 24670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b7352d143b46d9807164ca01d02efb: Generated new TSK 0
I20260928 02:45:49.459645 24670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:49.470482 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24583
2026-09-28T02:45:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:49 dist-test-slave-rqvb krb5kdc[24682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:49 dist-test-slave-rqvb krb5kdc[24682](info): set up 2 sockets
Sep 28 02:45:49 dist-test-slave-rqvb krb5kdc[24682](info): commencing operation
krb5kdc: starting...
W20260928 02:45:50.421864 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:50 dist-test-slave-rqvb krb5kdc[24682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:50.537375 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41543
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:41543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:50.643244 24695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:50.643498 24695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:50.643539 24695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:50.647487 24695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:50.647542 24695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:50.647559 24695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:50.647574 24695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:50.647588 24695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:50.652779 24695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:41543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:41543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:50.653757 24695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:50.654740 24695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:50.662758 24701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:50.662743 24703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:50.662762 24700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:50.663050 24695 server_base.cc:1034] running on GCE node
I20260928 02:45:50.663559 24695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:50.664651 24695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:50.665822 24695 hybrid_clock.cc:648] HybridClock initialized: now 1790563550665805 us; error 28 us; skew 500 ppm
Sep 28 02:45:50 dist-test-slave-rqvb krb5kdc[24682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563550, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:50.668619 24695 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:50.669544 24695 webserver.cc:460] Webserver started at http://127.2.123.62:45407/ using document root <none> and password file <none>
I20260928 02:45:50.670131 24695 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:50.670202 24695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:50.670408 24695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:50.672411 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "8c72de88a6e445d39dd9bbfa8b9e595a"
format_stamp: "Formatted at 2026-09-28 02:45:50 on dist-test-slave-rqvb"
server_key: "6d8c5a470c62ddcd7c265c6469838338"
server_key_iv: "92e90d40e2d6342795be6645047d9694"
server_key_version: "encryptionkey@0"
I20260928 02:45:50.672936 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "8c72de88a6e445d39dd9bbfa8b9e595a"
format_stamp: "Formatted at 2026-09-28 02:45:50 on dist-test-slave-rqvb"
server_key: "6d8c5a470c62ddcd7c265c6469838338"
server_key_iv: "92e90d40e2d6342795be6645047d9694"
server_key_version: "encryptionkey@0"
I20260928 02:45:50.676586 24695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 02:45:50.679005 24710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:50.680229 24695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:45:50.680338 24695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "8c72de88a6e445d39dd9bbfa8b9e595a"
format_stamp: "Formatted at 2026-09-28 02:45:50 on dist-test-slave-rqvb"
server_key: "6d8c5a470c62ddcd7c265c6469838338"
server_key_iv: "92e90d40e2d6342795be6645047d9694"
server_key_version: "encryptionkey@0"
I20260928 02:45:50.680451 24695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:50.684918 24695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:50.687839 24695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:50.687999 24695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:50.694661 24695 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:41543
I20260928 02:45:50.694700 24762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:41543 every 8 connection(s)
I20260928 02:45:50.695609 24695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:50.698300 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:50.703024 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24695
I20260928 02:45:50.703130 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:50.703363 2540 external_mini_cluster.cc:1428] Setting key 47a6706d2648f7e7560c764e43a9a912
I20260928 02:45:50.704013 24763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a: Bootstrap starting.
I20260928 02:45:50.706604 24763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:50.707486 24763 log.cc:824] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:50.708782 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:50.704695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35069 (local address 127.2.123.62:41543)
0928 02:45:50.705125 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:50.705145 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:50.705191 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:45:50.706030 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:50.706054 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:50.706397 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:50.706793 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:50.707482 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:50.707488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:50.707503 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:50.707515 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:50.707576 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:50.707605 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:50.707719 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:50.707804 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:45:50.707950 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260928 02:45:50.709510 24763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a: No bootstrap required, opened a new log
I20260928 02:45:50.711314 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:50.713564 24763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c72de88a6e445d39dd9bbfa8b9e595a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41543 } }
I20260928 02:45:50.713841 24763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:50.713907 24763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c72de88a6e445d39dd9bbfa8b9e595a, State: Initialized, Role: FOLLOWER
I20260928 02:45:50.714366 24763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c72de88a6e445d39dd9bbfa8b9e595a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41543 } }
I20260928 02:45:50.714509 24763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:50.714617 24763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:50.714730 24763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:50.715204 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:50.713651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45747 (local address 127.2.123.62:41543)
0928 02:45:50.713777 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:50.713782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:50.713871 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:45:50.713995 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:50.713998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:50.714035 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:50.714131 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:50.714807 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:50.714811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:50.714813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:45:50.714826 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:50.714853 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:45:50.714864 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:50.714963 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:50.715087 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:45:50.715116 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:45:50.715771 24763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c72de88a6e445d39dd9bbfa8b9e595a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41543 } }
I20260928 02:45:50.716089 24763 leader_election.cc:302] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c72de88a6e445d39dd9bbfa8b9e595a; no voters:
I20260928 02:45:50.716367 24763 leader_election.cc:288] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:50.716450 24773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:50.716672 24773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [term 1 LEADER]: Becoming Leader. State: Replica: 8c72de88a6e445d39dd9bbfa8b9e595a, State: Running, Role: LEADER
I20260928 02:45:50.717007 24773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c72de88a6e445d39dd9bbfa8b9e595a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41543 } }
I20260928 02:45:50.717473 24763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:50.718487 24774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c72de88a6e445d39dd9bbfa8b9e595a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c72de88a6e445d39dd9bbfa8b9e595a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41543 } } }
I20260928 02:45:50.718601 24774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:50.718932 24782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:50.718891 24775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c72de88a6e445d39dd9bbfa8b9e595a. Latest consensus state: current_term: 1 leader_uuid: "8c72de88a6e445d39dd9bbfa8b9e595a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c72de88a6e445d39dd9bbfa8b9e595a" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 41543 } } }
I20260928 02:45:50.719072 24775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:50.721923 24782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:50.733564 24782 catalog_manager.cc:1347] Generated new cluster ID: 38985bb6846e4f8db901c6098b7c8e52
I20260928 02:45:50.733721 24782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:50.740700 24782 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:50.740999 24782 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:50.750066 24782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c72de88a6e445d39dd9bbfa8b9e595a: Generated new TSK 0
I20260928 02:45:50.750741 24782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:45:50.760717 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24695
2026-09-28T02:45:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1290 ms)
[----------] 18 tests from AuthTokenIssuingTest (23471 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:50 dist-test-slave-rqvb krb5kdc[24794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:50 dist-test-slave-rqvb krb5kdc[24794](info): set up 2 sockets
Sep 28 02:45:50 dist-test-slave-rqvb krb5kdc[24794](info): commencing operation
krb5kdc: starting...
W20260928 02:45:51.710019 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:51 dist-test-slave-rqvb krb5kdc[24794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:51.825331 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:37819
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:37819,127.2.123.61:37513,127.2.123.60:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:51.934401 24807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:45:51.934669 24807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:51.934691 24807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:51.934731 24807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:51.938671 24807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:51.938728 24807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:51.938746 24807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:51.938761 24807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:51.938776 24807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:51.943990 24807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:37819,127.2.123.61:37513,127.2.123.60:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:37819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:51.944950 24807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:51.945888 24807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:51.954090 24812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:51.954180 24813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:51.954202 24807 server_base.cc:1034] running on GCE node
W20260928 02:45:51.954102 24815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:51.954946 24807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:51.956040 24807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:51.957221 24807 hybrid_clock.cc:648] HybridClock initialized: now 1790563551957196 us; error 36 us; skew 500 ppm
Sep 28 02:45:51 dist-test-slave-rqvb krb5kdc[24794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:51.959846 24807 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:51.960729 24807 webserver.cc:460] Webserver started at http://127.2.123.62:38901/ using document root <none> and password file <none>
I20260928 02:45:51.961230 24807 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:51.961285 24807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:51.961447 24807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:51.963459 24807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "d0e457638c1f4cf89096ea5e60884fee"
format_stamp: "Formatted at 2026-09-28 02:45:51 on dist-test-slave-rqvb"
server_key: "a0fe7a32d0a96111c05aabbf15a3ac1d"
server_key_iv: "e34e65d10da41cf494e3af630a5bfdb6"
server_key_version: "encryptionkey@0"
I20260928 02:45:51.963958 24807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "d0e457638c1f4cf89096ea5e60884fee"
format_stamp: "Formatted at 2026-09-28 02:45:51 on dist-test-slave-rqvb"
server_key: "a0fe7a32d0a96111c05aabbf15a3ac1d"
server_key_iv: "e34e65d10da41cf494e3af630a5bfdb6"
server_key_version: "encryptionkey@0"
I20260928 02:45:51.967630 24807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:45:51.970090 24822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:51.971323 24807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:45:51.971436 24807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "d0e457638c1f4cf89096ea5e60884fee"
format_stamp: "Formatted at 2026-09-28 02:45:51 on dist-test-slave-rqvb"
server_key: "a0fe7a32d0a96111c05aabbf15a3ac1d"
server_key_iv: "e34e65d10da41cf494e3af630a5bfdb6"
server_key_version: "encryptionkey@0"
I20260928 02:45:51.971535 24807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:51.988888 24807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:51.991888 24807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:51.992065 24807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:51.998802 24807 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:37819
I20260928 02:45:51.999085 24874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:37819 every 8 connection(s)
I20260928 02:45:51.999753 24807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:52.001179 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24807
I20260928 02:45:52.001278 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:52.001520 2540 external_mini_cluster.cc:1428] Setting key 8ad45018fa834b3bea7081953f898637
I20260928 02:45:52.002523 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:52.006210 24875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.123.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.61@KRBTEST.COM" created.
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:52.021906 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.008561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37819 (local address 127.0.0.1:42750)
0928 02:45:52.009056 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.009102 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.010017 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.010750 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.010756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.011195 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.011484 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.011499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.012916 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.012920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.014083 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.014093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.014707 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.014712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.014837 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.015390 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.015411 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.017105 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.019198 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.019205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.019213 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.019429 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.019710 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.019712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.019714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.019820 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.020577 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.020581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.020698 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.020834 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.021100 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"spinlock_wait_cycles":7552,"thread_start_us":150,"threads_started":1}
I20260928 02:45:52.021908 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.008560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.2.123.62:37819)
0928 02:45:52.008962 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.008977 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.009206 (+ 229us) server_negotiation.cc:407] Connection header received
0928 02:45:52.010318 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.010323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.010506 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.010719 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.011639 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.012794 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.014186 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.014552 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.014687 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.017255 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.017262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.017275 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.017299 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.019102 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.019531 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.019535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.019536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.019639 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.019998 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.020001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.020002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.020451 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.020600 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.020972 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.021168 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":268,"thread_start_us":107,"threads_started":1}
I20260928 02:45:52.024341 24875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:52.024992 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.024764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36500)
0928 02:45:52.024861 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.024934 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:45:52.025163 24825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab.
Entry for principal kudu/127.2.123.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab.
I20260928 02:45:52.026036 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.025822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36512)
0928 02:45:52.025907 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.025953 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 02:45:52.026170 24875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:37513: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
I20260928 02:45:52.026723 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.61:37513
--webserver_interface=127.2.123.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab
--principal=kudu/127.2.123.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:37819,127.2.123.61:37513,127.2.123.60:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:45:52.075408 24875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } attempt: 1
I20260928 02:45:52.076309 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.076018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36522)
0928 02:45:52.076166 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.076230 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:45:52.076968 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.076790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36526)
0928 02:45:52.076873 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.076915 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 02:45:52.077095 24875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:37513: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
W20260928 02:45:52.133701 24882 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:45:52.134056 24882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:52.134095 24882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:52.134159 24882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:52.138253 24882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:52.138316 24882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:52.138343 24882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:52.138363 24882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:52.138384 24882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:52.143718 24882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:37819,127.2.123.61:37513,127.2.123.60:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab
--principal=kudu/127.2.123.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.61:37513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.123.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:52.144846 24882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:52.145263 24875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } attempt: 2
I20260928 02:45:52.146143 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.145764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36542)
0928 02:45:52.145909 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.146055 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:45:52.146155 24882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:45:52.146958 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.146748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36548)
0928 02:45:52.146859 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.146904 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:45:52.147106 24875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:37513: Network error: Client connection negotiation failed: client connection to 127.2.123.61:37513: connect: Connection refused (error 111)
W20260928 02:45:52.155196 24887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:52.155186 24888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:52.155314 24882 server_base.cc:1034] running on GCE node
W20260928 02:45:52.155201 24890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:52.156033 24882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:52.157168 24882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:52.158347 24882 hybrid_clock.cc:648] HybridClock initialized: now 1790563552158306 us; error 51 us; skew 500 ppm
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:52.161118 24882 init.cc:377] Logged in from keytab as kudu/127.2.123.61@KRBTEST.COM (short username kudu)
I20260928 02:45:52.162019 24882 webserver.cc:460] Webserver started at http://127.2.123.61:35771/ using document root <none> and password file <none>
I20260928 02:45:52.162540 24882 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:52.162609 24882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:52.162782 24882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:52.164770 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/data/instance:
uuid: "b13d7be441f74b66a31fe7ae34b537ef"
format_stamp: "Formatted at 2026-09-28 02:45:52 on dist-test-slave-rqvb"
server_key: "6735352edfd3fe33d04faa95c2e38517"
server_key_iv: "58c1a56d8c0d7f0e1de131dbeb09eb69"
server_key_version: "encryptionkey@0"
I20260928 02:45:52.165289 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/wal/instance:
uuid: "b13d7be441f74b66a31fe7ae34b537ef"
format_stamp: "Formatted at 2026-09-28 02:45:52 on dist-test-slave-rqvb"
server_key: "6735352edfd3fe33d04faa95c2e38517"
server_key_iv: "58c1a56d8c0d7f0e1de131dbeb09eb69"
server_key_version: "encryptionkey@0"
I20260928 02:45:52.168882 24882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:45:52.171307 24897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:52.172731 24882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:52.172837 24882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/wal
uuid: "b13d7be441f74b66a31fe7ae34b537ef"
format_stamp: "Formatted at 2026-09-28 02:45:52 on dist-test-slave-rqvb"
server_key: "6735352edfd3fe33d04faa95c2e38517"
server_key_iv: "58c1a56d8c0d7f0e1de131dbeb09eb69"
server_key_version: "encryptionkey@0"
I20260928 02:45:52.172948 24882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:52.192075 24882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:52.195276 24882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:52.195439 24882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:52.202251 24882 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.61:37513
I20260928 02:45:52.202507 24949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.61:37513 every 8 connection(s)
I20260928 02:45:52.203312 24882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
I20260928 02:45:52.206138 24950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:52.209827 24950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:52.212785 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24882
I20260928 02:45:52.212904 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-1/wal/instance
I20260928 02:45:52.213171 2540 external_mini_cluster.cc:1428] Setting key 4d1f1f04f5f9d419fa6580bfe8c9af3d
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.60@KRBTEST.COM" created.
I20260928 02:45:52.223131 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.211717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.2.123.62:37819)
0928 02:45:52.211858 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.211862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.212736 (+ 874us) server_negotiation.cc:407] Connection header received
0928 02:45:52.213874 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.213876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.213922 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.214072 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.215391 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.215904 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.217067 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.217298 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.217388 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.219666 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.219669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.219672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.219697 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.221019 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.221523 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.221526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.221527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.221604 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.221991 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.221994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.221996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.222199 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.222289 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.222884 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.223007 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:45:52.224170 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.212079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37819 (local address 127.0.0.1:42756)
0928 02:45:52.212551 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.212610 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.213660 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.214243 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.214251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.214685 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.215249 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.215263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.216021 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.216023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.216958 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.216967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.217434 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.217439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.217601 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.218156 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.218172 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.219509 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.221129 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.221136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.221151 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.221394 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.221713 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.221716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.221718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.221880 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.222288 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.222295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.222584 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.222798 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.223151 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260928 02:45:52.225520 24950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:45:52.237119 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.226039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36550)
0928 02:45:52.226200 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.226213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.226281 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.226785 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.226788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.226964 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.227070 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.227077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.228148 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.228152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.229000 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.229011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.229605 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.229609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.229712 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.230219 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.230232 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.233138 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.235398 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.235401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.235402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.235629 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.236021 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.236025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.236027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.236124 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.236687 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.236690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.236772 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.236858 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.236962 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 02:45:52.237391 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.226123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.2.123.61:37513)
0928 02:45:52.226408 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.226419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.226438 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:45:52.226473 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.226476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.226653 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.226767 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.227287 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.228014 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.229118 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.229493 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.229572 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.233286 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.233296 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.233308 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.233335 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.235262 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.235753 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.235758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.235759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.235902 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.236233 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.236236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.236237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.236533 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.236714 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.237061 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.237231 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab.
Entry for principal kudu/127.2.123.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab.
I20260928 02:45:52.238255 24875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } attempt: 3
I20260928 02:45:52.238857 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.60:38305
--webserver_interface=127.2.123.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab
--principal=kudu/127.2.123.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:37819,127.2.123.61:37513,127.2.123.60:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:45:52.239482 24950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:52.240154 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.239883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59318)
0928 02:45:52.239996 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.240082 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 02:45:52.240402 24901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
I20260928 02:45:52.241318 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.241089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59322)
0928 02:45:52.241217 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.241260 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 02:45:52.241439 24950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:38305: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:45:52.247329 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.238770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.2.123.61:37513)
0928 02:45:52.240203 (+ 1433us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.240206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.240219 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:45:52.240252 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.240255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.240294 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.240363 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.240828 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.241382 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.242234 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.242496 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.242605 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.244616 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.244619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.244622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.244645 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.245869 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.246314 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.246317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.246319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.246385 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.246616 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.246619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.246621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.246776 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.246856 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.247102 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.247197 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371}
I20260928 02:45:52.247330 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.238730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36556)
0928 02:45:52.238870 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.238885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.238974 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.240405 (+ 1431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.240408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.240597 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.240699 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.240707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.241511 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.241514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.242130 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.242140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.242618 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.242622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.242732 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.243142 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.243155 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.244491 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.245991 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.245995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.245997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.246227 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.246463 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.246466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.246468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.246543 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.246860 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.246863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.246953 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.247044 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.247174 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 02:45:52.248308 24875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:52.248935 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.248732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59330)
0928 02:45:52.248834 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.248880 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:45:52.249552 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.249330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59340)
0928 02:45:52.249451 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.249495 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:45:52.249653 24875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:38305: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
I20260928 02:45:52.280813 24875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } attempt: 1
I20260928 02:45:52.281746 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.281397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59348)
0928 02:45:52.281584 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.281657 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:45:52.282543 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.282289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59356)
0928 02:45:52.282423 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.282478 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 02:45:52.282688 24875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:38305: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
I20260928 02:45:52.290688 24950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } attempt: 1
I20260928 02:45:52.291538 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.291234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59364)
0928 02:45:52.291383 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.291453 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:45:52.292268 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.292060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59380)
0928 02:45:52.292158 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.292207 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:45:52.292397 24950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:38305: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
W20260928 02:45:52.348788 24956 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:45:52.349072 24956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:52.349094 24956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:52.349130 24956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:52.353289 24956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:52.353351 24956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:52.353369 24956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:52.353384 24956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:52.353397 24956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:52.357842 24875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } attempt: 2
I20260928 02:45:52.358685 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.358421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59394)
0928 02:45:52.358553 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.358613 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:45:52.359346 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.359152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59408)
0928 02:45:52.359245 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.359291 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:45:52.359570 24875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:38305: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
I20260928 02:45:52.360535 24950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } attempt: 2
I20260928 02:45:52.360808 24956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:37819,127.2.123.61:37513,127.2.123.60:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab
--principal=kudu/127.2.123.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.60:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.123.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:52.361135 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.360912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59410)
0928 02:45:52.361024 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.361074 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:45:52.361829 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.361593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59424)
0928 02:45:52.361724 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.361771 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 02:45:52.361976 24950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:38305: Network error: Client connection negotiation failed: client connection to 127.2.123.60:38305: connect: Connection refused (error 111)
I20260928 02:45:52.362098 24956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:52.363101 24956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:52.371165 24963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:52.371161 24962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:52.371187 24965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:52.371381 24956 server_base.cc:1034] running on GCE node
I20260928 02:45:52.372069 24956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:52.373178 24956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:52.374338 24956 hybrid_clock.cc:648] HybridClock initialized: now 1790563552374313 us; error 35 us; skew 500 ppm
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:52.377130 24956 init.cc:377] Logged in from keytab as kudu/127.2.123.60@KRBTEST.COM (short username kudu)
I20260928 02:45:52.378031 24956 webserver.cc:460] Webserver started at http://127.2.123.60:42073/ using document root <none> and password file <none>
I20260928 02:45:52.378592 24956 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:52.378659 24956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:52.378860 24956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:52.380949 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/data/instance:
uuid: "3ad7c26664224a75aa7b6f439f87df59"
format_stamp: "Formatted at 2026-09-28 02:45:52 on dist-test-slave-rqvb"
server_key: "c6adda3296b89b6a4fb7987343ba5697"
server_key_iv: "c8de444ac11ac8744232c57fb8e1de81"
server_key_version: "encryptionkey@0"
I20260928 02:45:52.381456 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/wal/instance:
uuid: "3ad7c26664224a75aa7b6f439f87df59"
format_stamp: "Formatted at 2026-09-28 02:45:52 on dist-test-slave-rqvb"
server_key: "c6adda3296b89b6a4fb7987343ba5697"
server_key_iv: "c8de444ac11ac8744232c57fb8e1de81"
server_key_version: "encryptionkey@0"
I20260928 02:45:52.385070 24956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:45:52.387415 24972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:52.388574 24956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:52.388684 24956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/wal
uuid: "3ad7c26664224a75aa7b6f439f87df59"
format_stamp: "Formatted at 2026-09-28 02:45:52 on dist-test-slave-rqvb"
server_key: "c6adda3296b89b6a4fb7987343ba5697"
server_key_iv: "c8de444ac11ac8744232c57fb8e1de81"
server_key_version: "encryptionkey@0"
I20260928 02:45:52.388775 24956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:52.403844 24956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:52.406810 24956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:52.406980 24956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:52.413307 24956 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.60:38305
I20260928 02:45:52.413565 25024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.60:38305 every 8 connection(s)
I20260928 02:45:52.414238 24956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
I20260928 02:45:52.415234 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 24956
I20260928 02:45:52.415342 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563278472902-2540-0/minicluster-data/master-2/wal/instance
I20260928 02:45:52.415593 2540 external_mini_cluster.cc:1428] Setting key ec87f018bc92b140659db25969907cbd
I20260928 02:45:52.417198 25025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:52.420843 25025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:52.429492 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.416878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.2.123.60:38305)
0928 02:45:52.417306 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.417323 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.417357 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:45:52.418090 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.418109 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.418426 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.418741 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.419552 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.420559 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.421424 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.421879 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.422047 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.424742 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.424749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.424760 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.424790 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.426541 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.426917 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.426923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.426928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.427074 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.427349 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.427351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.427352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.427729 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.427905 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.428223 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.428484 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":140,"threads_started":1}
I20260928 02:45:52.431645 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.422369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:37819 (local address 127.0.0.1:42764)
0928 02:45:52.422841 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.422886 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.423005 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.423294 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.423298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.423648 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.423804 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.423811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.424675 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.424679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.425544 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.425566 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.426019 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.426022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.426129 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.426517 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.426531 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.427883 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.430120 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.430127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.430131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.430328 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.430618 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.430620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.430622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.430712 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.431073 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.431079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.431190 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.431319 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.431480 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":129,"threads_started":1}
I20260928 02:45:52.431676 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.422428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.2.123.62:37819)
0928 02:45:52.422548 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.422552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.422977 (+ 425us) server_negotiation.cc:407] Connection header received
0928 02:45:52.423094 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.423097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.423150 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.423232 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.423932 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.424550 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.425674 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.425893 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.426017 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.428013 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.428015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.428017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.428040 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.430028 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.430415 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.430418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.430420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.430512 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.430794 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.430796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.430797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.430955 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.431026 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.431404 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.431503 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:45:52.432914 25025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:45:52.440012 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.431815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.2.123.62:37819)
0928 02:45:52.432103 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.432107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.432119 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:45:52.432149 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.432152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.432188 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.432250 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.432783 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.433344 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.434138 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.434334 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.434414 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.436663 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.436667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.436670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.436694 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.438228 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.438657 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.438660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.438662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.438739 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.439011 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.439014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.439016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.439294 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.439488 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.439666 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.439785 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
I20260928 02:45:52.440972 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.433374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:37513 (local address 127.0.0.1:36572)
0928 02:45:52.433580 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.433593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.433667 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.433868 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.433871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.434083 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.434171 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.434176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.434922 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.434925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.435428 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.435435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.435977 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.435980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.436053 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.436451 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.436462 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.437869 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.439581 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.439585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.439587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.439821 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.440091 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.440094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.440096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.440190 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.440528 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.440531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.440614 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.440699 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.440820 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260928 02:45:52.441012 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.433469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.2.123.61:37513)
0928 02:45:52.433580 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.433585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.433634 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:45:52.433740 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.433743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.433785 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.433848 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.434288 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.434833 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.435536 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.435837 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.435947 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.438074 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.438077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.438079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.438106 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.439459 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.439920 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.439924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.439926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.440002 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.440273 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.440275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.440276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.440436 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.440517 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.440763 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.440865 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:45:52.441674 25025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:45:52.449802 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.441406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.2.123.61:37513)
0928 02:45:52.441489 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.441493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.441505 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:45:52.441549 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.441551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.441594 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.441663 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.442381 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.443042 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.444117 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.444370 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.444462 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.446994 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.446997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.446999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.447019 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.448268 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.448687 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.448690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.448691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.448765 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.448989 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.448992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.449024 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.449213 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.449425 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.449551 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.449656 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:45:52.450080 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.442207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59448)
0928 02:45:52.442288 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.442300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.442363 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.442542 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.442545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.442712 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.442794 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.442799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.444246 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.444249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.444851 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.444859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.445369 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.445371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.445423 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.445788 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.445797 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.447159 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.448674 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.448677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.448679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.448904 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.449145 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.449148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.449150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.449226 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.449581 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.449583 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.449667 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.449791 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.449903 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 02:45:52.450160 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.442256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.2.123.60:38305)
0928 02:45:52.442363 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.442368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.442381 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:45:52.442424 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.442427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.442462 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.442515 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.442880 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.444157 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.444941 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.445271 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.445381 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.447262 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.447266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.447269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.447292 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.448567 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.448985 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.448988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.448990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.449066 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.449305 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.449307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.449309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.449495 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.449577 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.449856 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.449990 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:45:52.453013 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:52.453822 25025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59: Bootstrap starting.
I20260928 02:45:52.453980 24950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } attempt: 3
I20260928 02:45:52.454423 24849 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d0e457638c1f4cf89096ea5e60884fee"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42772
I20260928 02:45:52.456642 25025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:52.457504 25025 log.cc:824] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:45:52.458688 24875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } attempt: 3
I20260928 02:45:52.459833 25025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59: No bootstrap required, opened a new log
I20260928 02:45:52.461488 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.454338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59456)
0928 02:45:52.454476 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.454488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.454567 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.454762 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.454765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.454908 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.454979 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.454984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.455648 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.455651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.456193 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.456201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.456594 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.456597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.456674 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.457073 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.457087 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.458266 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.460046 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.460049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.460051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.460287 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.460551 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.460553 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.460555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.460644 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.461006 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.461009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.461101 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.461205 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.461325 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 02:45:52.461508 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.454371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.2.123.60:38305)
0928 02:45:52.454463 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.454467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.454539 (+ 72us) server_negotiation.cc:407] Connection header received
0928 02:45:52.454639 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.454642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.454680 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.454740 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.455079 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.455533 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.456292 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.456503 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.456593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.458388 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.458391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.458394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.458416 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.459952 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.460391 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.460394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.460395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.460470 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.460728 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.460731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.460733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.460908 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.460987 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.461262 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.461368 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 02:45:52 dist-test-slave-rqvb krb5kdc[24794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:45:52.463901 25025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } }
I20260928 02:45:52.464151 25025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad7c26664224a75aa7b6f439f87df59, State: Initialized, Role: FOLLOWER
I20260928 02:45:52.464497 25025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } }
I20260928 02:45:52.464913 24950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef: Bootstrap starting.
I20260928 02:45:52.465068 25033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.465337 25033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:52.465822 25025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:52.467691 24950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:52.467707 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.459033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:38305 (local address 127.0.0.1:59472)
0928 02:45:52.459204 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:52.459216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:52.459282 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:52.459911 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:52.459914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:52.460076 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:52.460164 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.460169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.461081 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.461084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.461798 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:52.461808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.462343 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.462347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:52.462441 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.462901 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:52.462914 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:52.464307 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:52.466219 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.466222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.466224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.466460 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.466748 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:52.466751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:52.466753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:52.466838 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:52.467252 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:52.467256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:52.467349 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 02:45:52.467425 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:45:52.467553 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 02:45:52.468466 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.459099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.2.123.60:38305)
0928 02:45:52.459618 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.459652 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.459666 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:45:52.459727 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.459730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.459771 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.460244 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.460320 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.460967 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.461897 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.462232 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.462327 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.464470 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.464474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.464477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.464499 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.466093 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.466575 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.466578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.466580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.466654 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.466941 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.466944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.466946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.467146 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.467966 (+ 820us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.468057 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.468164 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":315,"threads_started":1}
I20260928 02:45:52.468575 24950 log.cc:824] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:52.470650 24950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef: No bootstrap required, opened a new log
W20260928 02:45:52.470803 25044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:52.470866 25044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:52.471282 24875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee: Bootstrap starting.
I20260928 02:45:52.469791 24956 master_runner.cc:186] Error getting master registration for 127.2.123.62:37819: OK, instance_id { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" instance_seqno: 1790563551996927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:45:52.472424 24956 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:45:52.473728 24875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:52.474445 24950 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } }
I20260928 02:45:52.474666 24950 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13d7be441f74b66a31fe7ae34b537ef, State: Initialized, Role: FOLLOWER
I20260928 02:45:52.474738 24875 log.cc:824] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:52.474999 24950 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } }
I20260928 02:45:52.475567 25048 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.475771 25048 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:52.476292 24950 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:52.476891 24875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee: No bootstrap required, opened a new log
I20260928 02:45:52.479802 24875 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } }
I20260928 02:45:52.480014 24875 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e457638c1f4cf89096ea5e60884fee, State: Initialized, Role: FOLLOWER
I20260928 02:45:52.480335 24875 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } }
W20260928 02:45:52.481333 25059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:52.481423 25059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:52.481570 24875 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:52.481671 24849 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:45:52.481776 24849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:52.483105 24849 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } }
I20260928 02:45:52.479610 24882 master_runner.cc:186] Error getting master registration for 127.2.123.62:37819: OK, instance_id { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" instance_seqno: 1790563551996927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:45:52.483338 24882 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:45:52.483628 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.483753 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:52.484014 24849 leader_election.cc:288] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [CANDIDATE]: Term 1 election: Requested vote from peers b13d7be441f74b66a31fe7ae34b537ef (127.2.123.61:37513), 3ad7c26664224a75aa7b6f439f87df59 (127.2.123.60:38305)
I20260928 02:45:52.484504 24924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0e457638c1f4cf89096ea5e60884fee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b13d7be441f74b66a31fe7ae34b537ef"
I20260928 02:45:52.484740 24924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:52.484840 24999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0e457638c1f4cf89096ea5e60884fee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ad7c26664224a75aa7b6f439f87df59"
I20260928 02:45:52.485067 24999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:52.485950 24924 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e457638c1f4cf89096ea5e60884fee in term 1.
I20260928 02:45:52.486549 24999 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e457638c1f4cf89096ea5e60884fee in term 1.
I20260928 02:45:52.486402 24825 leader_election.cc:302] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b13d7be441f74b66a31fe7ae34b537ef, d0e457638c1f4cf89096ea5e60884fee; no voters:
I20260928 02:45:52.487560 25061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:52.487751 25061 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [term 1 LEADER]: Becoming Leader. State: Replica: d0e457638c1f4cf89096ea5e60884fee, State: Running, Role: LEADER
I20260928 02:45:52.488055 25061 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } }
I20260928 02:45:52.490577 25077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e457638c1f4cf89096ea5e60884fee. Latest consensus state: current_term: 1 leader_uuid: "d0e457638c1f4cf89096ea5e60884fee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.490686 25077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: This master's current role is: LEADER
W20260928 02:45:52.493790 24828 tablet.cc:2369] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:52.498648 25079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:52.501780 25079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:52.503275 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.494961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.2.123.62:37819)
0928 02:45:52.495135 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.495139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.495150 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:45:52.495180 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.495183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.495220 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.495287 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.496168 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.496767 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.497631 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.497847 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.497932 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.499405 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.499409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.499411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.499436 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.500958 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.501440 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.501444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.501446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.501536 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.501816 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.501820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.501821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.502122 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.502241 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.503012 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.503126 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 02:45:52.512770 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.498305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.2.123.60:38305)
0928 02:45:52.498422 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.498426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.502902 (+ 4476us) server_negotiation.cc:407] Connection header received
0928 02:45:52.504425 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.504429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.504481 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.504530 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.505400 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.506193 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.507279 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.507656 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.507730 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.509170 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.509175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.509178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.509205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.510878 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.511327 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.511330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.511333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.511475 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.511773 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.511777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.511779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.511985 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.512117 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.512487 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.512607 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:45:52.516837 24999 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [term 1 FOLLOWER]: Refusing update from remote peer d0e457638c1f4cf89096ea5e60884fee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:52.516918 24924 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [term 1 FOLLOWER]: Refusing update from remote peer d0e457638c1f4cf89096ea5e60884fee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:52.517539 25077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:52.518031 25077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:52.521311 25048 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e457638c1f4cf89096ea5e60884fee. Latest consensus state: current_term: 1 leader_uuid: "d0e457638c1f4cf89096ea5e60884fee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.521431 25048 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:52.521675 25033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e457638c1f4cf89096ea5e60884fee. Latest consensus state: current_term: 1 leader_uuid: "d0e457638c1f4cf89096ea5e60884fee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.521782 25033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:52.523604 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:52.494881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.2.123.61:37513)
0928 02:45:52.494995 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.494999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.502325 (+ 7326us) server_negotiation.cc:407] Connection header received
0928 02:45:52.502503 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.502506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.502568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.502637 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.503618 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.504342 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.505618 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:52.505944 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:52.506035 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:52.507673 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:52.507677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:52.507679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:52.507706 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:52.509403 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.510118 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.510121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.510123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.510203 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:52.522072 (+ 11869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:52.522076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:52.522078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:52.522333 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:52.522473 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:52.523300 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 02:45:52.523432 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:45:52.525420 25077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0e457638c1f4cf89096ea5e60884fee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.525542 25077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:52.525540 25048 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0e457638c1f4cf89096ea5e60884fee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.525636 25048 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:52.525996 25084 mvcc.cc:205] Tried to move back new op lower bound from 7334148311102357504 to 7334148310994006016. Current Snapshot: MvccSnapshot[applied={T|T < 7334148311102357504}]
I20260928 02:45:52.527490 25086 mvcc.cc:205] Tried to move back new op lower bound from 7334148311102357504 to 7334148310994006016. Current Snapshot: MvccSnapshot[applied={T|T < 7334148311102357504}]
I20260928 02:45:52.530681 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0e457638c1f4cf89096ea5e60884fee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.530793 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:52.531565 25033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0e457638c1f4cf89096ea5e60884fee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e457638c1f4cf89096ea5e60884fee" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 37819 } } peers { permanent_uuid: "b13d7be441f74b66a31fe7ae34b537ef" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 37513 } } peers { permanent_uuid: "3ad7c26664224a75aa7b6f439f87df59" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 38305 } } }
I20260928 02:45:52.531683 25033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:52.531908 25085 mvcc.cc:205] Tried to move back new op lower bound from 7334148311102357504 to 7334148310994006016. Current Snapshot: MvccSnapshot[applied={T|T < 7334148311102357504}]
I20260928 02:45:52.541373 25079 catalog_manager.cc:1347] Generated new cluster ID: 784981801b6840efb00c1429456193bf
I20260928 02:45:52.541452 25079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:52.557826 25079 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:52.559207 25079 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:52.577279 25079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0e457638c1f4cf89096ea5e60884fee: Generated new TSK 0
I20260928 02:45:52.578271 25079 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:45:52.614305 24878 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.610442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.2.123.62:37819)
0928 02:45:52.610599 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.610604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.612431 (+ 1827us) server_negotiation.cc:407] Connection header received
0928 02:45:52.613122 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.613126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.613179 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.613253 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.614162 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42792: BlockingRecv error: recv got EOF from 127.0.0.1:42792 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:45:52.615080 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.610896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.2.123.61:37513)
0928 02:45:52.611005 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.611009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.612626 (+ 1617us) server_negotiation.cc:407] Connection header received
0928 02:45:52.613396 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.613400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.613450 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.613497 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.614965 (+ 1468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36610: BlockingRecv error: recv got EOF from 127.0.0.1:36610 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:45:52.616170 25032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.611317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.2.123.60:38305)
0928 02:45:52.611425 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.611429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.612937 (+ 1508us) server_negotiation.cc:407] Connection header received
0928 02:45:52.615421 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.615425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.615475 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.615526 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.616048 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59500: BlockingRecv error: recv got EOF from 127.0.0.1:59500 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 02:45:52.620702 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.619258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.2.123.61:37513)
0928 02:45:52.619356 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.619360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.619824 (+ 464us) server_negotiation.cc:407] Connection header received
0928 02:45:52.620018 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.620021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.620092 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.620142 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.620608 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36618: BlockingRecv error: recv got EOF from 127.0.0.1:36618 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 02:45:52.625499 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.624092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.2.123.61:37513)
0928 02:45:52.624188 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.624192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.624692 (+ 500us) server_negotiation.cc:407] Connection header received
0928 02:45:52.624866 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.624869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.624914 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.624960 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.625414 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36626: BlockingRecv error: recv got EOF from 127.0.0.1:36626 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 02:45:52.631175 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.629642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.2.123.61:37513)
0928 02:45:52.629738 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.629742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.630282 (+ 540us) server_negotiation.cc:407] Connection header received
0928 02:45:52.630489 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.630492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.630541 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.630602 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.631073 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36628: BlockingRecv error: recv got EOF from 127.0.0.1:36628 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 02:45:52.638942 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.637330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.2.123.61:37513)
0928 02:45:52.637457 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.637461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.638010 (+ 549us) server_negotiation.cc:407] Connection header received
0928 02:45:52.638199 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.638202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.638253 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.638300 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.638837 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36644: BlockingRecv error: recv got EOF from 127.0.0.1:36644 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 02:45:52.651378 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.649481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.2.123.61:37513)
0928 02:45:52.649619 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.649624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.650284 (+ 660us) server_negotiation.cc:407] Connection header received
0928 02:45:52.650509 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.650513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.650584 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.650646 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.651258 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36656: BlockingRecv error: recv got EOF from 127.0.0.1:36656 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 02:45:52.671412 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.669748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.2.123.61:37513)
0928 02:45:52.669864 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.669868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.670478 (+ 610us) server_negotiation.cc:407] Connection header received
0928 02:45:52.670677 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.670680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.670733 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.670782 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.671293 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36658: BlockingRecv error: recv got EOF from 127.0.0.1:36658 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:45:52.706768 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.705694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.2.123.61:37513)
0928 02:45:52.705828 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.705833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.705929 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:45:52.706097 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.706100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.706163 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.706267 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.706650 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36660: BlockingRecv error: recv got EOF from 127.0.0.1:36660 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:45:52.773741 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.772618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.2.123.61:37513)
0928 02:45:52.772775 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.772780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.772907 (+ 127us) server_negotiation.cc:407] Connection header received
0928 02:45:52.773055 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.773059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.773113 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.773182 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.773621 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36670: BlockingRecv error: recv got EOF from 127.0.0.1:36670 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 02:45:52.904661 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:52.903629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.2.123.61:37513)
0928 02:45:52.903760 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:52.903763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:52.903948 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:45:52.904016 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:52.904019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:52.904075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:52.904159 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:52.904565 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36678: BlockingRecv error: recv got EOF from 127.0.0.1:36678 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 02:45:53.164232 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:53.163119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.2.123.61:37513)
0928 02:45:53.163281 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:53.163286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:53.163345 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:45:53.163536 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:53.163539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:53.163587 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:53.163658 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:53.164127 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36684: BlockingRecv error: recv got EOF from 127.0.0.1:36684 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:45:53.472121 25044 catalog_manager.cc:1259] Loaded cluster ID: 784981801b6840efb00c1429456193bf
I20260928 02:45:53.472208 25044 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59: loading cluster ID for follower catalog manager: success
I20260928 02:45:53.473853 25044 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59: acquiring CA information for follower catalog manager: success
I20260928 02:45:53.475041 25044 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ad7c26664224a75aa7b6f439f87df59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:45:53.482609 25059 catalog_manager.cc:1259] Loaded cluster ID: 784981801b6840efb00c1429456193bf
I20260928 02:45:53.482684 25059 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef: loading cluster ID for follower catalog manager: success
I20260928 02:45:53.484254 25059 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef: acquiring CA information for follower catalog manager: success
I20260928 02:45:53.485316 25059 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b13d7be441f74b66a31fe7ae34b537ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:45:53.682289 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:53.678340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.2.123.61:37513)
0928 02:45:53.678586 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:53.678591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:53.678730 (+ 139us) server_negotiation.cc:407] Connection header received
0928 02:45:53.678886 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:53.678888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:53.679010 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:53.679103 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:53.679376 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.679899 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.680783 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.681011 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.681100 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:53.681412 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:53.681842 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:53.682017 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:53.682085 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:45:53.682142 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260928 02:45:53.688946 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:53.685270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.2.123.60:38305)
0928 02:45:53.685488 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:53.685491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:53.685571 (+ 80us) server_negotiation.cc:407] Connection header received
0928 02:45:53.685699 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:53.685701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:53.685813 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:53.685906 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:53.686140 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.686658 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.687484 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.687716 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.687795 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:53.688137 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:53.688588 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:53.688690 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:53.688738 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:45:53.688788 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260928 02:45:53.695659 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:53.692030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.2.123.61:37513)
0928 02:45:53.692185 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:53.692189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:53.692254 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:45:53.692347 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:53.692350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:53.692424 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:53.692491 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:53.692812 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.693623 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.694593 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.694808 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.694892 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:53.695144 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:53.695228 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:53.695299 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:53.695467 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:45:53.695522 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:45:53.696527 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:53.691853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.2.123.60:38305)
0928 02:45:53.691953 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:53.691957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:53.692516 (+ 559us) server_negotiation.cc:407] Connection header received
0928 02:45:53.692638 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:53.692642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:53.692683 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:53.692754 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:53.692974 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.693724 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.694990 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.695248 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.695331 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:53.696145 (+ 814us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:53.696211 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:53.696268 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:53.696360 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:45:53.696408 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:45:53.696682 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:53.692226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.2.123.62:37819)
0928 02:45:53.692438 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:53.692441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:53.692471 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:45:53.692584 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:53.692587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:53.692749 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:53.692890 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:53.693109 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.693656 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.694973 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:53.695269 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:53.695443 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:53.695685 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:53.696021 (+ 336us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:53.696445 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:53.696479 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 02:45:53.696537 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
I20260928 02:45:53.702252 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24807
I20260928 02:45:53.707468 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24882
I20260928 02:45:53.712167 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 24956
2026-09-28T02:45:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2952 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:53 dist-test-slave-rqvb krb5kdc[25174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:53 dist-test-slave-rqvb krb5kdc[25174](info): set up 2 sockets
Sep 28 02:45:53 dist-test-slave-rqvb krb5kdc[25174](info): commencing operation
krb5kdc: starting...
W20260928 02:45:54.674252 2540 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/test-user.keytab.
Sep 28 02:45:54 dist-test-slave-rqvb krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
Entry for principal kudu/127.2.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab.
I20260928 02:45:54.792991 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45359
--webserver_interface=127.2.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:45359,127.2.123.61:43787,127.2.123.60:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:54.901604 25187 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:45:54.901888 25187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:54.901911 25187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:54.901988 25187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:54.906018 25187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:54.906088 25187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:54.906107 25187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:54.906121 25187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:54.906136 25187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:54.911405 25187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:45359,127.2.123.61:43787,127.2.123.60:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.62.keytab
--principal=kudu/127.2.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.62:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:54.912447 25187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:54.913462 25187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:54.921999 25193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:54.922039 25187 server_base.cc:1034] running on GCE node
W20260928 02:45:54.921999 25192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:54.922286 25195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:54.922997 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:54.924407 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:54.925592 25187 hybrid_clock.cc:648] HybridClock initialized: now 1790563554925561 us; error 48 us; skew 500 ppm
Sep 28 02:45:54 dist-test-slave-rqvb krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:54.928619 25187 init.cc:377] Logged in from keytab as kudu/127.2.123.62@KRBTEST.COM (short username kudu)
I20260928 02:45:54.929497 25187 webserver.cc:460] Webserver started at http://127.2.123.62:39739/ using document root <none> and password file <none>
I20260928 02:45:54.930083 25187 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:54.930208 25187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:54.930418 25187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:54.932492 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/data/instance:
uuid: "eeccad5f84c94496b0d50b487fa35b43"
format_stamp: "Formatted at 2026-09-28 02:45:54 on dist-test-slave-rqvb"
server_key: "ef3b3c53ca83504abfdeb88430693622"
server_key_iv: "59fb63d779e39e77507e1aa83a395b59"
server_key_version: "encryptionkey@0"
I20260928 02:45:54.933001 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/wal/instance:
uuid: "eeccad5f84c94496b0d50b487fa35b43"
format_stamp: "Formatted at 2026-09-28 02:45:54 on dist-test-slave-rqvb"
server_key: "ef3b3c53ca83504abfdeb88430693622"
server_key_iv: "59fb63d779e39e77507e1aa83a395b59"
server_key_version: "encryptionkey@0"
I20260928 02:45:54.936667 25187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:45:54.939229 25202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:54.940466 25187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:54.940587 25187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/wal
uuid: "eeccad5f84c94496b0d50b487fa35b43"
format_stamp: "Formatted at 2026-09-28 02:45:54 on dist-test-slave-rqvb"
server_key: "ef3b3c53ca83504abfdeb88430693622"
server_key_iv: "59fb63d779e39e77507e1aa83a395b59"
server_key_version: "encryptionkey@0"
I20260928 02:45:54.940682 25187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:54.952672 25187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:54.955490 25187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:54.955665 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:54.962376 25187 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.62:45359
I20260928 02:45:54.962651 25254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.62:45359 every 8 connection(s)
I20260928 02:45:54.963337 25187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:54.966547 25255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:54.969152 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 25187
I20260928 02:45:54.969266 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-0/wal/instance
I20260928 02:45:54.969592 2540 external_mini_cluster.cc:1428] Setting key c5111679e0a97a6095f492ae1a431c08
I20260928 02:45:54.970757 25255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.123.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.61@KRBTEST.COM" created.
Sep 28 02:45:54 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:54.987214 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:54.973126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45359 (local address 127.0.0.1:53086)
0928 02:45:54.973747 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:54.973794 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:54.974629 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:54.975474 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:54.975479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:54.975808 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:54.976088 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:54.976103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:54.977809 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:54.977812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:54.978676 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:54.978686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:54.979410 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:54.979413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:54.979522 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:54.980151 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:54.980168 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:54.981738 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:54.984031 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:54.984039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:54.984048 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:54.984279 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:54.984664 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:54.984667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:54.984668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:54.984787 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:54.985606 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:54.985611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:54.985742 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 02:45:54.985896 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:45:54.986426 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":140,"spinlock_wait_cycles":1920,"thread_start_us":147,"threads_started":1}
I20260928 02:45:54.987217 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:54.973126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.2.123.62:45359)
0928 02:45:54.973689 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:54.973707 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:54.973905 (+ 198us) server_negotiation.cc:407] Connection header received
0928 02:45:54.974962 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:54.974969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:54.975237 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:54.975454 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:54.976238 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:54.977665 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:54.978805 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:54.979258 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:54.979385 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:54.981925 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:54.981931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:54.981943 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:54.982006 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:54.983902 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:54.984390 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:54.984397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:54.984398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:54.984537 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:54.984880 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:54.984883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:54.984885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:54.985466 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:54.985616 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:54.986034 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 02:45:54.986320 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":138,"threads_started":1}
I20260928 02:45:54.989782 25255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:54.990531 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:54.990246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54156)
0928 02:45:54.990381 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:54.990465 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 02:45:54.990731 25204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
I20260928 02:45:54.991511 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:54.991332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54162)
0928 02:45:54.991417 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:54.991460 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:45:54.991657 25255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:43787: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab.
Entry for principal kudu/127.2.123.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab.
I20260928 02:45:54.996981 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.61:43787
--webserver_interface=127.2.123.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab
--principal=kudu/127.2.123.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:45359,127.2.123.61:43787,127.2.123.60:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:45:55.040926 25255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } attempt: 1
I20260928 02:45:55.041724 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.041439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54166)
0928 02:45:55.041577 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.041647 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:45:55.042569 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.042359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54174)
0928 02:45:55.042462 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.042510 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:45:55.042804 25255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:43787: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
W20260928 02:45:55.107842 25262 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:45:55.108132 25262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:55.108155 25262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:55.108191 25262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:45:55.110966 25255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } attempt: 2
I20260928 02:45:55.111759 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.111468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54180)
0928 02:45:55.111613 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.111684 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:45:55.112394 25262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:55.112450 25262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:55.112468 25262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:55.112484 25262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:55.112512 25262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:55.112514 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.112299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54190)
0928 02:45:55.112397 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.112443 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 02:45:55.112654 25255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.61:43787: Network error: Client connection negotiation failed: client connection to 127.2.123.61:43787: connect: Connection refused (error 111)
I20260928 02:45:55.118065 25262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:45359,127.2.123.61:43787,127.2.123.60:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.61.keytab
--principal=kudu/127.2.123.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.61:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.123.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:55.119089 25262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:55.120066 25262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:55.128319 25270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:55.128417 25262 server_base.cc:1034] running on GCE node
W20260928 02:45:55.128353 25268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:55.128301 25267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:55.129176 25262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:55.130286 25262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:55.131441 25262 hybrid_clock.cc:648] HybridClock initialized: now 1790563555131420 us; error 47 us; skew 500 ppm
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563555, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:55.134138 25262 init.cc:377] Logged in from keytab as kudu/127.2.123.61@KRBTEST.COM (short username kudu)
I20260928 02:45:55.134960 25262 webserver.cc:460] Webserver started at http://127.2.123.61:41737/ using document root <none> and password file <none>
I20260928 02:45:55.135452 25262 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:55.135516 25262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:55.135684 25262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:55.137669 25262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/data/instance:
uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac"
format_stamp: "Formatted at 2026-09-28 02:45:55 on dist-test-slave-rqvb"
server_key: "b513f84dce5787191d1865b4e2112f1f"
server_key_iv: "f8c3e78a3505b3934fada660fe3ac0d3"
server_key_version: "encryptionkey@0"
I20260928 02:45:55.138236 25262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/wal/instance:
uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac"
format_stamp: "Formatted at 2026-09-28 02:45:55 on dist-test-slave-rqvb"
server_key: "b513f84dce5787191d1865b4e2112f1f"
server_key_iv: "f8c3e78a3505b3934fada660fe3ac0d3"
server_key_version: "encryptionkey@0"
I20260928 02:45:55.141848 25262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:45:55.144376 25277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:55.145732 25262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:55.145843 25262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/wal
uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac"
format_stamp: "Formatted at 2026-09-28 02:45:55 on dist-test-slave-rqvb"
server_key: "b513f84dce5787191d1865b4e2112f1f"
server_key_iv: "f8c3e78a3505b3934fada660fe3ac0d3"
server_key_version: "encryptionkey@0"
I20260928 02:45:55.145926 25262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:55.171861 25262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:55.175113 25262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:55.175274 25262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:55.182214 25262 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.61:43787
I20260928 02:45:55.182468 25329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.61:43787 every 8 connection(s)
I20260928 02:45:55.183174 25262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/data/info.pb
I20260928 02:45:55.185864 25330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:55.189445 25330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:55.193265 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 25262
I20260928 02:45:55.193403 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-1/wal/instance
I20260928 02:45:55.193691 2540 external_mini_cluster.cc:1428] Setting key 9f39d267e47dad3337324f9ec83b0535
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563555, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.123.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.123.60@KRBTEST.COM" created.
I20260928 02:45:55.203213 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.191268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.2.123.62:45359)
0928 02:45:55.191412 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.191416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.192198 (+ 782us) server_negotiation.cc:407] Connection header received
0928 02:45:55.193114 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.193162 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.193222 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.193312 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.194522 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.195271 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.196356 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.196583 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.196680 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.199079 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.199083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.199085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.199114 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.201076 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.201627 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.201630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.201631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.201720 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.202063 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.202066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.202067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.202257 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.202339 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.202914 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.203044 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:45:55.203794 25255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } attempt: 3
I20260928 02:45:55.204056 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.191609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45359 (local address 127.0.0.1:53096)
0928 02:45:55.192046 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.192093 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:55.192898 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:55.193498 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:55.193506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:55.193887 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:55.194381 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.194393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.195393 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.195396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.196239 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.196248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.196725 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.196729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.196884 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.197518 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:55.197535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:55.198919 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:55.201228 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.201234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.201245 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.201512 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.201804 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.201806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.201807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.201936 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.202377 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:55.202383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:55.202656 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 02:45:55.202845 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 02:45:55.203158 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260928 02:45:55.206586 25330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563555, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:45:55.214808 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.204209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54204)
0928 02:45:55.204459 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.204475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:55.204552 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:55.205463 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:55.205466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:55.205672 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:55.205791 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.205798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.207349 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.207353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.208062 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.208073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.208773 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.208777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.208879 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.209324 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:55.209337 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:55.210605 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:55.213187 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.213191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.213192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.213374 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.213707 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.213710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.213711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.213788 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.214343 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:55.214346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:55.214448 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:45:55.214567 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 02:45:55.214687 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260928 02:45:55.214983 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.204703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.2.123.61:43787)
0928 02:45:55.205045 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.205059 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.205079 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:45:55.205117 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.205123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.205331 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.205469 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.205896 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.206796 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.208177 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.208666 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.208761 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.210765 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.210775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.210789 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.210815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.213081 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.213472 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.213477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.213478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.213571 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.213872 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.213875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.213877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.214201 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.214372 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.214658 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.214795 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":242,"thread_start_us":141,"threads_started":1}
I20260928 02:45:55.215794 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.207056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54214)
0928 02:45:55.207196 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.207207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:55.207259 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:55.207647 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:55.207650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:55.207830 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:55.207951 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.207958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.208885 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.208888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.209664 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.209674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.210271 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.210275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.210370 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.210756 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:55.210769 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:55.212054 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:55.214440 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.214443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.214445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.214644 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.214887 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.214889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.214891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.214955 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.215300 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:55.215303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:55.215377 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 02:45:55.215460 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:45:55.215584 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 02:45:55.215806 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.207150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.2.123.61:43787)
0928 02:45:55.207434 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.207441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.207452 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:45:55.207480 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.207482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.207518 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.207609 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.208080 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.208769 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.209788 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.210139 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.210232 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.212241 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.212246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.212249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.212273 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.214347 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.214735 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.214737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.214739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.214804 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.215043 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.215045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.215047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.215217 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.215301 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.215509 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.215646 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
I20260928 02:45:55.216802 25330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:55.217008 25255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:55.217442 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.217238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:50960)
0928 02:45:55.217327 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.217391 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:45:55.217628 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.217409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:50962)
0928 02:45:55.217522 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.217570 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:45:55.217628 25281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
I20260928 02:45:55.218358 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.218088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:50972)
0928 02:45:55.218239 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.218311 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 02:45:55.218498 25255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:45273: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
I20260928 02:45:55.218539 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.218342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:50982)
0928 02:45:55.218443 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.218489 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:45:55.218636 25330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:45273: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.123.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab.
Entry for principal kudu/127.2.123.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab.
I20260928 02:45:55.221210 2540 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
/tmp/dist-test-taskCSnLDv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.60:45273
--webserver_interface=127.2.123.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab
--principal=kudu/127.2.123.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.123.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.123.62:45359,127.2.123.61:43787,127.2.123.60:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:45:55.249670 25255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } attempt: 1
I20260928 02:45:55.250613 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.250265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:50986)
0928 02:45:55.250426 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.250492 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:45:55.251313 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.251109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:50992)
0928 02:45:55.251207 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.251255 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:45:55.251430 25255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:45273: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
I20260928 02:45:55.267905 25330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } attempt: 1
I20260928 02:45:55.268833 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.268481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51000)
0928 02:45:55.268654 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.268737 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:45:55.269624 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.269391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51012)
0928 02:45:55.269507 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.269561 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 02:45:55.269780 25330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:45273: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
I20260928 02:45:55.326592 25255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } attempt: 2
I20260928 02:45:55.327481 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.327150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51024)
0928 02:45:55.327338 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.327401 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:45:55.328146 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.327950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51032)
0928 02:45:55.328048 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.328093 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:45:55.328310 25255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:45273: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
W20260928 02:45:55.331882 25338 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:45:55.332165 25338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:55.332208 25338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:55.332273 25338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:55.336331 25338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:55.336397 25338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:55.336422 25338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:55.336444 25338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:55.336472 25338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:55.337944 25330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } attempt: 2
I20260928 02:45:55.338728 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.338435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51034)
0928 02:45:55.338595 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.338656 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:45:55.339411 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.339188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51036)
0928 02:45:55.339304 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.339355 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:45:55.339530 25330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.123.60:45273: Network error: Client connection negotiation failed: client connection to 127.2.123.60:45273: connect: Connection refused (error 111)
I20260928 02:45:55.342097 25338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.123.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.123.62:45359,127.2.123.61:43787,127.2.123.60:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/krb5kdc/kudu_127.2.123.60.keytab
--principal=kudu/127.2.123.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.123.60:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.123.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:45:55.343114 25338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:55.344137 25338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:55.352425 25343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:55.352432 25344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:55.352421 25346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:55.352837 25338 server_base.cc:1034] running on GCE node
I20260928 02:45:55.353336 25338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:55.354445 25338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:55.355625 25338 hybrid_clock.cc:648] HybridClock initialized: now 1790563555355606 us; error 35 us; skew 500 ppm
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563555, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:55.358242 25338 init.cc:377] Logged in from keytab as kudu/127.2.123.60@KRBTEST.COM (short username kudu)
I20260928 02:45:55.359038 25338 webserver.cc:460] Webserver started at http://127.2.123.60:44269/ using document root <none> and password file <none>
I20260928 02:45:55.359552 25338 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:55.359620 25338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:55.359815 25338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:45:55.361778 25338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/data/instance:
uuid: "eab4231c6e354648b7c00363c2b5f40b"
format_stamp: "Formatted at 2026-09-28 02:45:55 on dist-test-slave-rqvb"
server_key: "6bdc224f9dcb5a7d07f1f4687db7ecb3"
server_key_iv: "2d31074cafbbba43356f34c9e700bb7a"
server_key_version: "encryptionkey@0"
I20260928 02:45:55.362337 25338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/wal/instance:
uuid: "eab4231c6e354648b7c00363c2b5f40b"
format_stamp: "Formatted at 2026-09-28 02:45:55 on dist-test-slave-rqvb"
server_key: "6bdc224f9dcb5a7d07f1f4687db7ecb3"
server_key_iv: "2d31074cafbbba43356f34c9e700bb7a"
server_key_version: "encryptionkey@0"
I20260928 02:45:55.365866 25338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:55.368284 25353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:55.369691 25338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:55.369812 25338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/data,/tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/wal
uuid: "eab4231c6e354648b7c00363c2b5f40b"
format_stamp: "Formatted at 2026-09-28 02:45:55 on dist-test-slave-rqvb"
server_key: "6bdc224f9dcb5a7d07f1f4687db7ecb3"
server_key_iv: "2d31074cafbbba43356f34c9e700bb7a"
server_key_version: "encryptionkey@0"
I20260928 02:45:55.369922 25338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:55.404868 25338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:55.407924 25338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:55.408106 25338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:55.414502 25338 rpc_server.cc:307] RPC server started. Bound to: 127.2.123.60:45273
I20260928 02:45:55.414748 25405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.123.60:45273 every 8 connection(s)
I20260928 02:45:55.415426 25338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/data/info.pb
I20260928 02:45:55.417719 2540 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu as pid 25338
I20260928 02:45:55.417836 2540 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCSnLDv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563278472902-2540-0/minicluster-data/master-2/wal/instance
I20260928 02:45:55.418113 25406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:55.418169 2540 external_mini_cluster.cc:1428] Setting key 41f60865b7e170572ddbde42579dc699
I20260928 02:45:55.422657 25406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:45:55.427446 25255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } attempt: 3
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563555, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:55.430672 25330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } attempt: 3
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.62@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:45:55.433687 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.419363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.2.123.60:45273)
0928 02:45:55.419715 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.419729 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.419763 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:45:55.420584 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.420601 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.420917 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.421188 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.422113 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.423116 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.424204 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.424862 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.425038 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.427695 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.427702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.427713 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.427744 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.429672 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.430193 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.430199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.430205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.430335 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.430633 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.430637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.430639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.431131 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.432034 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.432196 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.432467 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260928 02:45:55.434401 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.424445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.2.123.62:45359)
0928 02:45:55.424561 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.424565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.424837 (+ 272us) server_negotiation.cc:407] Connection header received
0928 02:45:55.424955 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.424958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.425008 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.425086 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.425702 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.426491 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.427419 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.427706 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.427792 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.430268 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.430272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.430274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.430298 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.432549 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.433025 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.433028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.433030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.433107 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.433399 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.433402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.433404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.433631 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.433704 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.434151 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.434257 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:45:55.434420 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.424386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.62:45359 (local address 127.0.0.1:53110)
0928 02:45:55.424726 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.424763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:55.424870 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:55.425142 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:55.425147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:55.425460 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:55.425582 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.425588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.426633 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.426637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.427315 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.427324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.427805 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.427809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.428141 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.428591 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:55.428607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:55.430158 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:55.432660 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.432667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.432672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.432921 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.433222 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.433225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.433227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.433306 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.433783 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:55.433788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:55.433897 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 02:45:55.434080 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 02:45:55.434259 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260928 02:45:55.436096 25406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:55.436509 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.427833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51056)
0928 02:45:55.427936 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.427947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:55.428004 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:55.428795 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:55.428805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:55.428972 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:55.429064 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.429071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.430050 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.430053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.430786 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.430799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.431315 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.431319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.431392 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.431791 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:55.431805 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:55.433228 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:55.435077 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.435080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.435082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.435332 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.435622 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.435625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.435627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.435700 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.436063 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:55.436066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:55.436152 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 02:45:55.436254 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:45:55.436374 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 02:45:55.437224 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.428072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.2.123.60:45273)
0928 02:45:55.428375 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.428384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.428397 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:45:55.428429 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.428432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.428471 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.428568 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.429196 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.429935 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.430885 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.431185 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.431588 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.433364 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.433367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.433369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.433392 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.434960 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.435434 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.435437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.435439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.435529 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.435783 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.435786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.435788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.435939 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.436019 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.436952 (+ 933us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.437058 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563555, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.61@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:45:55.440899 25255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43: Bootstrap starting.
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563555, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.62@KRBTEST.COM
I20260928 02:45:55.443464 25255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:55.444207 25255 log.cc:824] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:55.445034 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.436716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.2.123.61:43787)
0928 02:45:55.436825 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.436829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.437082 (+ 253us) server_negotiation.cc:407] Connection header received
0928 02:45:55.437252 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.437255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.437301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.437351 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.437840 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.438595 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.439543 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.439879 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.439957 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.441758 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.441762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.441764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.441790 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.443319 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.443775 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.443778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.443780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.443860 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.444162 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.444166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.444167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.444369 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.444436 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.444771 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.444900 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:45:55.445472 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.436492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.61:43787 (local address 127.0.0.1:54224)
0928 02:45:55.437024 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.437036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:55.437138 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:55.437472 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:55.437474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:55.437618 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:55.437714 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.437720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.438750 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.438753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.439446 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.439456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.440056 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.440059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.440128 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.440529 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:55.440542 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:55.441610 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:55.443438 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.443442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.443443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.443660 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.443966 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.443969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.443971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.444052 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.444502 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:55.444504 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:55.444583 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 02:45:55.445227 (+ 644us) client_negotiation.cc:240] Negotiation successful
0928 02:45:55.445347 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476}
I20260928 02:45:55.446154 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.436903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.2.123.62:45359)
0928 02:45:55.437018 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.437022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.437034 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:45:55.437063 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.437066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.437102 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.437438 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.438203 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.438774 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.439504 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.439822 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.439914 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.442396 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.442400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.442402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.442427 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.443930 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.444404 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.444408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.444409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.444494 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.444831 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.444834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.444836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.445125 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.445796 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.445871 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.446012 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:45:55.446444 25406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:45:55.447841 25255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43: No bootstrap required, opened a new log
I20260928 02:45:55.448633 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.431070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51058)
0928 02:45:55.431269 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.431282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:55.431351 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:55.432446 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:55.432450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:55.432625 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:55.432743 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.432750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.435623 (+ 2873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.435626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.436368 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.436379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.436912 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.436916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.436996 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.437406 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:55.437419 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:55.444960 (+ 7541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:55.447168 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.447172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.447174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.447409 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.447703 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.447707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.447708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.447796 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.448227 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:55.448230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:55.448313 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 02:45:55.448381 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:45:55.448492 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260928 02:45:55.449587 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.431158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.2.123.60:45273)
0928 02:45:55.432182 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.432190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.432203 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:45:55.432234 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.432236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.432291 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.432851 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.433109 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.435485 (+ 2376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.436490 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.436792 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.437188 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.445084 (+ 7896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.445087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.445090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.445112 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.446975 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.447510 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.447513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.447514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.447587 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.447939 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.447941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.447943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.448124 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.449261 (+ 1137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.449330 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.449433 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":95,"threads_started":1}
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563555, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.123.60@KRBTEST.COM for kudu/127.2.123.60@KRBTEST.COM
I20260928 02:45:55.452371 25255 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } }
I20260928 02:45:55.452610 25255 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeccad5f84c94496b0d50b487fa35b43, State: Initialized, Role: FOLLOWER
I20260928 02:45:55.452874 25330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac: Bootstrap starting.
I20260928 02:45:55.452942 25255 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } }
I20260928 02:45:55.453445 25414 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.453595 25414 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:55.454341 25255 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:55.455621 25330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:55.455686 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.446860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.123.60:45273 (local address 127.0.0.1:51070)
0928 02:45:55.447106 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:55.447118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:55.447211 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:55.447478 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:55.447481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:55.447618 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:55.447698 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.447703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.448567 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.448570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.449307 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:55.449314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.449880 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.449883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:55.449952 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.450703 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:55.450712 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:55.452110 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:55.453895 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.453898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.453900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.454211 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.454741 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:55.454744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:55.454745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:55.454824 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:55.455243 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:55.455246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:55.455330 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 02:45:55.455401 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:45:55.455523 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
Sep 28 02:45:55 dist-test-slave-rqvb krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.123.61@KRBTEST.COM
I20260928 02:45:55.456133 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.447023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.2.123.60:45273)
0928 02:45:55.447109 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.447113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.447166 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:45:55.447292 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.447295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.447329 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.447371 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.447790 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.448468 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.449405 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.449709 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.449774 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.452224 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.452232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.452234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.452259 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.453805 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.454315 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.454318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.454526 (+ 208us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.454614 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.454933 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.454936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.454938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.455139 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.455836 (+ 697us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.455906 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.456007 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:45:55.456588 25330 log.cc:824] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac: Log is configured to *not* fsync() on all Append() calls
W20260928 02:45:55.457059 25208 tablet.cc:2369] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:45:55.459276 25330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac: No bootstrap required, opened a new log
W20260928 02:45:55.459678 25425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:55.459743 25425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:55.459290 25187 master_runner.cc:186] Error getting master registration for 127.2.123.61:43787: OK, instance_id { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" instance_seqno: 1790563555180386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:45:55.462976 25187 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:45:55.463115 25406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b: Bootstrap starting.
I20260928 02:45:55.463187 25330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } }
I20260928 02:45:55.463407 25330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e87eedcd9ed4af784c3c7ad4c17b8ac, State: Initialized, Role: FOLLOWER
I20260928 02:45:55.463690 25330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } }
I20260928 02:45:55.464100 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.464277 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:55.465073 25330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:55.465727 25406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:55.466588 25406 log.cc:824] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:55.467724 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.447247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.2.123.61:43787)
0928 02:45:55.447338 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.447342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.447485 (+ 143us) server_negotiation.cc:407] Connection header received
0928 02:45:55.447644 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.447647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.447686 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.447730 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.448486 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.449152 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.450213 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.450501 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.450594 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.459613 (+ 9019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.459618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.459620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.459645 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.461160 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.461643 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.461647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.461648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.461731 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.462122 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.462126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.462128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.462351 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.462524 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.467437 (+ 4913us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.467575 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:45:55.470973 25406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b: No bootstrap required, opened a new log
W20260928 02:45:55.471213 25439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:55.471320 25439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:55.471774 2540 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:45:55.473825 25229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eeccad5f84c94496b0d50b487fa35b43"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53116
I20260928 02:45:55.474062 25229 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:45:55.474179 25229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:55.474843 25406 raft_consensus.cc:348] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } }
I20260928 02:45:55.475090 25406 raft_consensus.cc:729] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab4231c6e354648b7c00363c2b5f40b, State: Initialized, Role: FOLLOWER
I20260928 02:45:55.475474 25229 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } }
I20260928 02:45:55.475440 25406 consensus_queue.cc:260] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } }
I20260928 02:45:55.477149 25406 sys_catalog.cc:564] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:55.477202 25380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eeccad5f84c94496b0d50b487fa35b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eab4231c6e354648b7c00363c2b5f40b"
I20260928 02:45:55.477401 25380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:55.478713 25380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeccad5f84c94496b0d50b487fa35b43 in term 1.
I20260928 02:45:55.479326 25206 leader_election.cc:302] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eab4231c6e354648b7c00363c2b5f40b, eeccad5f84c94496b0d50b487fa35b43; no voters:
I20260928 02:45:55.479633 25414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:55.476701 25262 master_runner.cc:186] Error getting master registration for 127.2.123.60:45273: OK, instance_id { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" instance_seqno: 1790563555412733 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:45:55.479872 25414 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [term 1 LEADER]: Becoming Leader. State: Replica: eeccad5f84c94496b0d50b487fa35b43, State: Running, Role: LEADER
I20260928 02:45:55.479921 25262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:45:55.480201 25414 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } }
I20260928 02:45:55.480894 25303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eeccad5f84c94496b0d50b487fa35b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac"
I20260928 02:45:55.481109 25303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:55.482024 25414 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeccad5f84c94496b0d50b487fa35b43. Latest consensus state: current_term: 1 leader_uuid: "eeccad5f84c94496b0d50b487fa35b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.482127 25414 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:55.482396 25303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeccad5f84c94496b0d50b487fa35b43 in term 1.
W20260928 02:45:55.482681 25453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:45:55.482769 25453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:45:55.484747 25229 leader_election.cc:288] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [CANDIDATE]: Term 1 election: Requested vote from peers 7e87eedcd9ed4af784c3c7ad4c17b8ac (127.2.123.61:43787), eab4231c6e354648b7c00363c2b5f40b (127.2.123.60:45273)
I20260928 02:45:55.486191 25457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:55.487608 25457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:55.492640 25303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [term 1 FOLLOWER]: Refusing update from remote peer eeccad5f84c94496b0d50b487fa35b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:55.492656 25380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [term 1 FOLLOWER]: Refusing update from remote peer eeccad5f84c94496b0d50b487fa35b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:55.493312 25414 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:55.493731 25414 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:55.494103 25442 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.494248 25442 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:55.497387 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeccad5f84c94496b0d50b487fa35b43. Latest consensus state: current_term: 1 leader_uuid: "eeccad5f84c94496b0d50b487fa35b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.497484 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:55.497431 25442 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeccad5f84c94496b0d50b487fa35b43. Latest consensus state: current_term: 1 leader_uuid: "eeccad5f84c94496b0d50b487fa35b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.497545 25442 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:55.504343 25454 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeccad5f84c94496b0d50b487fa35b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.504462 25454 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:55.504563 25462 mvcc.cc:205] Tried to move back new op lower bound from 7334148323291385856 to 7334148323250089984. Current Snapshot: MvccSnapshot[applied={T|T < 7334148323291385856}]
I20260928 02:45:55.505299 25464 mvcc.cc:205] Tried to move back new op lower bound from 7334148323291385856 to 7334148323250089984. Current Snapshot: MvccSnapshot[applied={T|T < 7334148323291385856}]
I20260928 02:45:55.505431 25442 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeccad5f84c94496b0d50b487fa35b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.505525 25442 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:55.510362 25414 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeccad5f84c94496b0d50b487fa35b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.510473 25414 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:55.511531 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeccad5f84c94496b0d50b487fa35b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeccad5f84c94496b0d50b487fa35b43" member_type: VOTER last_known_addr { host: "127.2.123.62" port: 45359 } } peers { permanent_uuid: "7e87eedcd9ed4af784c3c7ad4c17b8ac" member_type: VOTER last_known_addr { host: "127.2.123.61" port: 43787 } } peers { permanent_uuid: "eab4231c6e354648b7c00363c2b5f40b" member_type: VOTER last_known_addr { host: "127.2.123.60" port: 45273 } } }
I20260928 02:45:55.511634 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:45:55.512104 25463 mvcc.cc:205] Tried to move back new op lower bound from 7334148323291385856 to 7334148323250089984. Current Snapshot: MvccSnapshot[applied={T|T < 7334148323291385856}]
I20260928 02:45:55.519188 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.499839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.2.123.62:45359)
0928 02:45:55.499946 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.499951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.503452 (+ 3501us) server_negotiation.cc:407] Connection header received
0928 02:45:55.504727 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.504731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.504788 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.504846 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.507824 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.508620 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.512524 (+ 3904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.512868 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.512939 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.515778 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.515782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.515785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.515812 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.517408 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.517870 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.517874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.517876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.517954 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.518274 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.518278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.518280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.518483 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.518601 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.518919 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.519029 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:45:55.521329 25457 catalog_manager.cc:1347] Generated new cluster ID: 371e5152cb924c55a4a7ab56419efcc9
I20260928 02:45:55.521394 25457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:55.525631 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.500639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.2.123.60:45273)
0928 02:45:55.500731 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.500735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.514005 (+ 13270us) server_negotiation.cc:407] Connection header received
0928 02:45:55.514190 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.514193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.514247 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.514298 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.515219 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.515904 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.517493 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.517841 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.517903 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.519833 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.519836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.519839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.519863 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.521388 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.522840 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.522843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.522845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.522939 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.523182 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.523185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.523187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.523363 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.523453 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.525361 (+ 1908us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.525485 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:45:55.537488 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:55.500310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.2.123.61:43787)
0928 02:45:55.500417 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.500421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.519743 (+ 19322us) server_negotiation.cc:407] Connection header received
0928 02:45:55.521983 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.521986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.522038 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.522089 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.526051 (+ 3962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.526790 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.528883 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:55.529213 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:55.529280 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:55.534076 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:55.534080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:55.534083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:55.534107 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:55.535708 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.536199 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.536203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.536204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.536294 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:55.536580 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:55.536584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:55.536585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:55.536805 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:55.536908 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:55.537231 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:45:55.537347 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:45:55.539656 25457 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:55.540858 25457 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:55.561040 25457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeccad5f84c94496b0d50b487fa35b43: Generated new TSK 0
I20260928 02:45:55.561942 25457 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:45:55.591636 25411 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.589095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.2.123.60:45273)
0928 02:45:55.589216 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.589221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.590540 (+ 1319us) server_negotiation.cc:407] Connection header received
0928 02:45:55.590756 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.590761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.590812 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.590863 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.591495 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51084: BlockingRecv error: recv got EOF from 127.0.0.1:51084 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 02:45:55.593036 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.589894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.2.123.61:43787)
0928 02:45:55.590025 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.590030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.592162 (+ 2132us) server_negotiation.cc:407] Connection header received
0928 02:45:55.592333 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.592337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.592385 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.592432 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.592920 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54248: BlockingRecv error: recv got EOF from 127.0.0.1:54248 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:45:55.594965 25258 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.593656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.2.123.62:45359)
0928 02:45:55.593786 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.593790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.594099 (+ 309us) server_negotiation.cc:407] Connection header received
0928 02:45:55.594276 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.594280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.594327 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.594373 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.594849 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53134: BlockingRecv error: recv got EOF from 127.0.0.1:53134 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:45:55.599159 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.597671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.2.123.61:43787)
0928 02:45:55.597771 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.597775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.598263 (+ 488us) server_negotiation.cc:407] Connection header received
0928 02:45:55.598484 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.598487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.598556 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.598603 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.599061 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54250: BlockingRecv error: recv got EOF from 127.0.0.1:54250 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 02:45:55.603811 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.602465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.2.123.61:43787)
0928 02:45:55.602556 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.602559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.603027 (+ 468us) server_negotiation.cc:407] Connection header received
0928 02:45:55.603191 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.603194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.603238 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.603283 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.603726 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54256: BlockingRecv error: recv got EOF from 127.0.0.1:54256 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 02:45:55.609246 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.607787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.2.123.61:43787)
0928 02:45:55.607879 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.607882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.608384 (+ 502us) server_negotiation.cc:407] Connection header received
0928 02:45:55.608559 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.608563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.608607 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.608666 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.609138 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54272: BlockingRecv error: recv got EOF from 127.0.0.1:54272 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 02:45:55.616806 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.615337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.2.123.61:43787)
0928 02:45:55.615455 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.615458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.615909 (+ 451us) server_negotiation.cc:407] Connection header received
0928 02:45:55.616076 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.616079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.616146 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.616197 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.616710 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54278: BlockingRecv error: recv got EOF from 127.0.0.1:54278 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 02:45:55.628894 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.627212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.2.123.61:43787)
0928 02:45:55.627328 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.627333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.627937 (+ 604us) server_negotiation.cc:407] Connection header received
0928 02:45:55.628132 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.628136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.628189 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.628240 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.628784 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54280: BlockingRecv error: recv got EOF from 127.0.0.1:54280 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:45:55.649382 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.647576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.2.123.61:43787)
0928 02:45:55.647720 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.647725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.648313 (+ 588us) server_negotiation.cc:407] Connection header received
0928 02:45:55.648546 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.648550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.648608 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.648677 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.649258 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54286: BlockingRecv error: recv got EOF from 127.0.0.1:54286 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 02:45:55.684924 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.683888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.2.123.61:43787)
0928 02:45:55.684025 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.684030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.684125 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:45:55.684255 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.684259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.684316 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.684419 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.684801 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54298: BlockingRecv error: recv got EOF from 127.0.0.1:54298 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 02:45:55.752094 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.750975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.2.123.61:43787)
0928 02:45:55.751094 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.751098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.751325 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:45:55.751403 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.751405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.751457 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.751547 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.751983 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54300: BlockingRecv error: recv got EOF from 127.0.0.1:54300 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 02:45:55.883814 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:55.882514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.2.123.61:43787)
0928 02:45:55.882648 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:55.882653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:55.882850 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:45:55.883028 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:55.883040 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:55.883098 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:55.883186 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:55.883697 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54314: BlockingRecv error: recv got EOF from 127.0.0.1:54314 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 02:45:56.142958 25336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:45:56.141921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.2.123.61:43787)
0928 02:45:56.142084 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:56.142089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:56.142193 (+ 104us) server_negotiation.cc:407] Connection header received
0928 02:45:56.142304 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:56.142307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:56.142362 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:56.142448 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:56.142841 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54328: BlockingRecv error: recv got EOF from 127.0.0.1:54328 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:45:56.472595 25439 catalog_manager.cc:1259] Loaded cluster ID: 371e5152cb924c55a4a7ab56419efcc9
I20260928 02:45:56.472692 25439 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac: loading cluster ID for follower catalog manager: success
I20260928 02:45:56.474488 25439 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac: acquiring CA information for follower catalog manager: success
I20260928 02:45:56.475616 25439 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e87eedcd9ed4af784c3c7ad4c17b8ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:45:56.483870 25453 catalog_manager.cc:1259] Loaded cluster ID: 371e5152cb924c55a4a7ab56419efcc9
I20260928 02:45:56.483940 25453 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b: loading cluster ID for follower catalog manager: success
I20260928 02:45:56.485558 25453 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b: acquiring CA information for follower catalog manager: success
I20260928 02:45:56.486696 25453 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eab4231c6e354648b7c00363c2b5f40b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:45:56.661064 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:56.657066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.2.123.61:43787)
0928 02:45:56.657313 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:56.657318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:56.657334 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:45:56.657510 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:56.657513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:56.657648 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:56.657752 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:56.658057 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.658649 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.659582 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.659816 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.659905 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:56.660230 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:56.660677 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:56.660793 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:56.660848 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:45:56.660909 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260928 02:45:56.667764 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:56.664049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51086 (local address 127.2.123.60:45273)
0928 02:45:56.664286 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:56.664290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:56.664305 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:45:56.664392 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:56.664394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:56.664505 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:56.664606 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:56.664831 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.665354 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.666333 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.666566 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.666673 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:56.666955 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:56.667394 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:56.667499 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:56.667562 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:45:56.667619 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
I20260928 02:45:56.673998 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:56.670635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.2.123.60:45273)
0928 02:45:56.670750 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:56.670754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:56.671015 (+ 261us) server_negotiation.cc:407] Connection header received
0928 02:45:56.671082 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:56.671085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:56.671131 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:56.671206 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:56.671416 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.672008 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.672845 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.673228 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.673322 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:56.673567 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:56.673633 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:56.673689 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:56.673797 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:45:56.673845 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:45:56.674700 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:56.670526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.2.123.61:43787)
0928 02:45:56.670641 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:56.670645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:56.671213 (+ 568us) server_negotiation.cc:407] Connection header received
0928 02:45:56.671397 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:56.671400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:56.671451 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:56.671518 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:56.671750 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.672515 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.673420 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.673753 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.673851 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:56.674240 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:56.674318 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:56.674361 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:56.674524 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:45:56.674580 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:45:56.675361 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:56.670515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.2.123.62:45359)
0928 02:45:56.671022 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:56.671027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:56.671040 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:45:56.671165 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:56.671168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:56.671325 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:56.671433 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:56.671802 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.672576 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.673815 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:56.674172 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:56.674266 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:56.674737 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:56.674998 (+ 261us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:56.675076 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:56.675173 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:45:56.675226 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":68,"threads_started":1}
I20260928 02:45:56.678267 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 25187
I20260928 02:45:56.683570 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 25262
I20260928 02:45:56.688012 2540 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCSnLDv/build/debug/bin/kudu with pid 25338
2026-09-28T02:45:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2975 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5928 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278206 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST